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EXECUTIVE SUMMARY 


This report is furnished to the Air Force Office of Scientific Research as the final 
deliverable for AFOSR Grant FA9550-12-1-0097, “The Role of Benevolence in Trust 
of Autonomous Systems.” This report provides a summary of the research and 
related activities performed with the support of this grant and key results, including 
pre-print copies of the peer-reviewed publications and related material. Without 
reliable and robust methods for assessing the trustworthiness of intelligent, 
autonomous systems, the issue of trust has become one of the most significant 
obstacles to broad use of autonomy technology by DoD and other agencies. 
However, the impact of the research described in this report supports creation of 


GERNNIETEEED The core objectives of this research were (1) a> 


eadway was 
achieved on key topics, including some notable results. Key accomplishments 
discussed in this report include: (1) the formulation of benevolence as a complex 
“belief structure” with antecedent beliefs having important semantic, temporal, 
causal and other interrelationships; (2) the mapping of a portion of this belief 
structure to measurable internal states of autonomous systems, thereby potentially 
creating new opportunities for assessment of trustworthiness of such systems; (3) 
the obtaining of empirical evidence in support of the proposition that previous 


psychological concepts of interpersonal human trust are applicable to trust in 
autonomous systems, including the role of personality and situation in modulating 
the role and importance of certain beliefs; (4) the creation of a theory of a "Human 
Social Interface” which, when expressed in systems engineering terms, provides 
guidance for machine portrayal of trust-related qualities in human-machine social 
interaction; (5) the design and implementation of a software prototype based on the 
Human Social Interface theory that provides a basis for future experimentation and 
evaluation. This project resulted in ia 
‘presentations in scientific venues, meetings with distinguished visitors, and other in 
support of technology transition opportunities within DoD and to industry. Many 
new research questions were generated and there remains considerable work to do 
to fully understand the role of benevolence with respect to intelligent autonomous 
systems. Overall, the theoretical foundation for trustworthiness of autonomous 
systems is immature and remains an important area of focus for multiple 
disciplines. 
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INTRODUCTION 


This report is furnished to the Air Force Office of Scientific Research as the final 
deliverable for AFOSR Grant FA9550-12-1-0097, “The Role of Benevolence in Trust 
of Autonomous Systems.” The research described here was performed between 
April 2012 and April 2015. The purpose of this report is to provide a summary of 
the research and related activities performed with the support of this grant and to 
summarize key results, including copies of the peer-reviewed publications and 
related documentary material. Please note that the publications found in the 
appendix here are author pre-print copies and may differ in some details from the 
published versions; the latter should be regarded as the scientific documentation of 
record. 


NEEDS OF THE US AIR FORCE 


The US Air Force forecasts the need to interact with and rely on increasingly 
intelligent autonomous systems. Without reliable and robust methods for assessing 
the trustworthiness of an autonomous system, the issue of trust has become one of 
the most significant obstacles to broad use of autonomy technology even as it 
rapidly matures. 


OBJECTIVES 


There were two primary objectives for the research, summarized as follows: 


1. Operationalize “benevolence” and understand how that quality contributes 
to well-calibrated trust of, and reliance upon, autonomous systems. 

2. Investigate measures and methods for portrayal of trust-related attributes 
such as "benevolence’ in the human-machine interface. 


APPROACH 


The overall approach of this project was to relate empirically discovered trust- 
related qualities to theoretical constructs on which to base computational methods 
for further experimentation and development. 


The research design was primarily qualitative and oriented towards establishing a 
theoretical framework for “benevolence” within which to examine the issues of trust 
and delegation to autonomous systems. In addition to theory development, this 
project included an exploratory survey to explore human attribution of trust-related 
qualities to autonomous systems and secondly, a human study to further refine the 
parameters of trust with a focus on the conditions under which an attribution of 
benevolence may occur. Analysis of the results, the publication and presentation of 
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study results in appropriate scientific venues, and this final report conclude the 
activities in this project. 


IMPACT 


The theory of trustworthiness in autonomous systems is immature. The scientific 
impact of operationalizing benevolence and component trust-related qualities using 
theoretical constructs from Cognitive Science and Al supports creation of 
computational methods to create a bridge to future engineering of trustworthy 
autonomous systems. 
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ACCOMPLISHMENTS 


The principal scientific accomplishments of this project are summarized below and 
discussed in detail in the following sections. 


The attributed quality of benevolence” to a candidate trustee (human or 
machine) was formulated as a construct consisting ofa rich set of component 
beliefs with complex interrelations. These component beliefs, or 
“antecedents” of benevolence, have each been the focus of previous studies of 
human interpersonal trust. The formulation of benevolence arising from this 
study revealed the importance of perception of agency and animacy 
(“liveness”) for autonomous systems. 


The project developed a semantic belief structure representation of trust 
qualities (component beliefs) for benevolence, including logical, temporal, 
causal, evidentiary relations and other dependencies among those beliefs and 
specified a preliminary mapping of those belief structure representations to 


facets of the internal state of autonomous systems. ThE objective OF devising, 


The project obtained empirical evidence that confirmed certain key abstract 
qualities of human interpersonal trustworthiness (ie., Competence, 
Predictability, Openness, Risk/Safety) are applicable to evaluation of 
autonomy trustworthiness. However, self-reports by study participants 
regarding the relative importance of trust-related qualities in the absence of 
specific context proved to be poor predictors of actual delegation decisions. 
The qualities most significantly related to evaluation of trustworthiness of an 
autonomous system, and their relative importance, varied by individual 
personality and situational factors (including, for example, perception and 
acceptance of risks of different types). 


‘engineering computational methods that portray anthropomorphic trust- 


related qualities in a human-machine cyber-physical interface. This 
formulation guided the design and programming of a software prototype for 
portrayal of trust-related qualities by a 

This novel software architecture features a hybrid reactive/deliberative 
control scheme that enables loose coupling and non-interference of social 
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and task behaviors, and is easily extended as new social interactive 
requirements for autonomous robots are defined. 


* The prototype Human Social Interface was tested in an immersive simulated 
environment designed to potentiate a heightened sense of danger. The 
simulation was designed to explore conditions under which attribution of 
benevolence might be important in a candidate autonomous robot 
application to urban rescue. A human study was designed, approved and 
implemented. 


Accomplishments are discussed in more detail in the following sections. Peer- 
reviewed publications (attached) provide the documentation of record. Two 
additional papers and an invention report are in preparation. 
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SUMMARY OF RESEARCH ACTIVITIES 


This section presents an overview of the key research activities performed under 
this grant. Results and other findings from these activities are described in the 
following section (“Summary of Findings”). 


THEORY DEVELOPMENT 


Theory development in this project focused on development of a social model of 
trust that extends the cognitive and affective nature of human interpersonal trust in 
ways that will ultimately provide guidance for the design and development of 
autonomous agents that have the ability to engender appropriate human-machine 
reliance and interdependency. Furthermore, we defined and explored the concept of 
a “Human Social Interface” in system engineering terms rather than psychological 
terms. This theoretical aspect of the project was oriented towards the objective of 
devising computation mechanisms for modulation of human beliefs by a socially 
competent intelligent autonomous agent. 


EXPLORATORY SURVEY RESEARCH 


Previous research in psychology and other fields guided the specification of 
candidate Belief Structures for trust in autonomous systems. These in turn informed 
the design of our survey research, robot simulation and human study protocol, and 
computational mechanisms. 


Sudesoftrat {TH 


Meta-Analysis 
Studies 


Beliefs Interrelationship 


Trust Beliefs Trust Belief 


Categories 


Belief Structures 


{ Survey Questions } 


Figure 1: Flow of Information from Previous Studies 


The trust-relevant internal state of a human agent is a complex cognitive and 
affective structure that references causal factors, attitudes, evaluations and 
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expectations centered on other agents (especially the potential “Trustee”), the 
situation, goals, and tasks. This state includes antecedent beliefs of the Trustor 
regarding a Trustee. Such beliefs may have complex interrelationships based on the 
relative contributions of evidence, causality and situational influences. To represent 
these beliefs and their interrelationships, we use the term “Belief Structure”. The 
word “structure” explicitly reminds us that the individual beliefs in each belief 
structure are complex, inter-related, conditional, and occasionally may even be 
contradictory. Our ultimate goal of a computational representation of a belief 
structure must be rich enough to capture this logical structure. 


An exploratory survey research study was performed to examine putative 
components of trust in autonomous agents. The study had three goals: 1) to assess 
which, if any, among a set of anthropomorphic beliefs derived from studies on 
human interpersonal trust are important to a human's decision to delegate to an 
intelligent, autonomous agent; 2) to determine the relative importance among such 
beliefs; 3) to explore whether the applicability or importance of those beliefs to a 
delegation decision vary in a systematic way by individual personality and/or 
situational factors. 


The survey was conducted online with a sample of participants drawn from a pool 
of autonomy subject matter experts and decision-makers. This group was targeted 
specifically due to the essential role they play in developing autonomy technology 
and, more importantly, in helping to make decisions on whether to create and field 
specific types of applications. Thus, their attitudes and beliefs regarding trust of 
autonomous systems have the potential for broad effects on all aspects of the 
autonomous system lifecycle. 


The survey design is summarized below. Please see the next section regarding 
findings and also refer to the publications located in the appendix for more 
information. The complete survey, as administered, was provided to Volkswagen 
Research of America, at their request, for use in research on the role of personality 
in trust of automobile autonomy. 


Survey Design: 


1. Participants ranked the absolute (not relative) importance of twenty-eight 
specific trust-related qualities of agents that span four Belief Structures defined by 
the theoretical portion of the project (Competence, Predictability, Openness, Safety). 
See Figure 2, below. 


2. Participants completed three standard personality survey instruments used in the 
social sciences: Big Five Inventory (BFI-40), Individual Innovativeness (II), and the 
Domain- Specific Risk Taking Scale (DOSPERT). 
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3. Participants were presented with six challenge scenarios that vary in terms of risk 
type and magnitude as well as relative emphasis on aspects of the four focus Belief 
Structures. Participants were forced to make a choice of whether to rely on human, 
autonomous system, or “other” agent to satisfy the needs presented in the scenario. 
(Descriptions of these scenarios are included in the Appendix.) 


4, Participants completed the Source Credibility inventory to explore the perceived 
“ethos” of the autonomous systems presented in the scenarios. This produced 
measures of perceived competence, goodwill, and overall trustworthiness. 


‘Table 2 Twenty Eight Hypothetical Trust-Related Qualities of Intelligent, Autonomous Agents” 


‘Category ‘Name ‘Quality Deseription 
Competence Capable The autonomous agent can achieve a desired result. 
Knowledge The autonomous agent has all the knowledge it needs todo its job. 
‘Accurate What the autonomous agent believes to be true is actually true. 
Skilled The autonomous agent possesses good methods for using its knowledge todo its task. 
Logical The autonomous agent reasons correctly according to logic. 
Heuristic When it cannot figure out something using logic, the autonomous agent can make good guesses. 
Corrective The autonomous agent recognizes gaps in its knowledge and tries to learn what it needs to know. 
‘Adaptive The autonomous agent learns to correct its mistakes, as well as to improve and maximize its capability. 
Predictability Expected The autonomous agent's behavior conforms to expectations. 
Purposeful ‘The autonomous agent purposefully acts to achieve goals. 
Helpful The autonomous agent will assist people, whenever itis possible. 
Directable The autonomous agent accepts and carries out orders. 
Reasonable The autonomous agent uses its knowledge and skill in expected ways. 
Safety Safe The autonomous agent's behavior will not harm humans or human interest, 
Limited Any incorrect behavior by the autonomous agent will not cause arm. 
Stable The autonomous agent fails gracefully and recovers from its failure promptly 
Ruled The autonomous agent adheres to obligations, principles, and rules. 
Correctable The autonomous agent can correct its own defects or they can be corrected by a human. 
Protective The autonomous agent recognizes and avoids harming humans’ interests. 
Favorable Given alternatives in what 10 do or how to do it, an autonomous agent will act ina way that is favorable to a 
‘human being who might be affected. 


Openness Visible What the autonomous agent is doing and how it works is easy to see and understand. 
Honest ‘The autonomous agent believes what it says, 
‘Transparent It is easy to inspect an autonomous agent. 


‘Communicative The autonomous agent communicates in a way that is easy to understand. 
Interactive The autonomous agent responds when you are trying to communicate with it. 
‘Attentive The autonomous agent is aware of communication between others nearby. 
Reactive The autonomous agent responds quickly to calls for attention. 
Disclosing _ The autonomous agent communicates truthfully and fully. 


® Quality Description is shown exactly as it appeared in the survey. 
Figure 2: Hypothetical Trust-Related Qualities 


PROTOCOL FOR STUDY OF ATTRIBUTION OF BENEVOLENCE 


The project developed an experimental protocol under which the attribution of 
benevolence by a person to an autonomous robot could be examined. As part of the 
apparatus for this study, a detailed immersive 3D simulation was constructed, 
including situation and environment conditions as well as emulated autonomous 
robots. The simulated situation and environment were designed to evoke a sense of 
danger in participants, potentiate “victim psychology,” and thereby establishing the 
conditions in which benevolence can become most important (See Figures 3 and 4). 
From the perspective of potential applications, the key question was whether the 
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perceived benevolence of an intelligent robot would increase cooperation and 
compliance with an offer of help from the robot (e.g. rescue). 


The emulation of robots within the simulation employed a prototype computational 
version of the Human Social Interface to control robot social behaviors during the 
study. In addition, software was developed for control of the trial. This included 
task sequence of events, a special interface for collection of pre- and post-task 
questionnaire data, and data acquisition within the simulated world. This software 
is Open Source and will be made available to other researchers. 


The study design and methodology was first presented at ACM/IEEE HAI-14 in 
Japan: 


* Atkinson, D.J. and Clark, M.H. Methodology for Study of Human-Robot Social 
Interaction in Dangerous Situations. In Proceedings of Human-Agent 
Interaction. DOI: 10.1145/2658861.2658871. ACM (2014). 


(See figures, next page) 
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‘otentiate fear of “predatory” attack 
Lighting creates dark shadowed areas 
Atmospheric diffusion limits distance clarity 
Visibility lines are obstructed 


Potentiate perception of danger with risk stimuli 
Worn out appearance, messy, signs of incivility (trash, graffiti) 
Presence of drums with warning signs of hazardous chemicals 
Prominent warning signs and fire alarms 


Figure 3: Simulated Environment Cues to Evoke Danger 
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Startling 


Figure 4: Participant POV in Simulated Warehouse Disaster 
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RESULTS AND FINDINGS 


This section presents an overview of results and other findings produced by the 
activities described in the previous section. It should be viewed as a guide to the full, 
detailed results as formally documented in peer-reviewed publications and related 
presentations. These are included in the Appendix to this report. Two additional 
publications are currently in preparation. 


BENEVOLENCE AND TRUST 


“Trustis not benevolence and benevolence is not trust The wo concepts have been 


frequently conflated in the history of social psychology. Attribution of 
trustworthiness and benevolence are both products of entangled cognitive and 
emotional processes (Cummings and Bromily 1996). A belief that a potential 
Trustee is benevolent entails many of the same beliefs that are also antecedents to 
trust. However, some of the unique characteristics of benevolence, as opposed to 
trustworthiness in general, suggest that we must be careful to consider the 
constructs individually. 


Believing that a Trustee is benevolent requires antecedent beliefs about the Trustee, 
such as those regarding motivation, dispositions and intentions, predictability, 
competence, and more. Furthermore, these antecedents may have complex 
interrelationships based on the relative contributions of evidence, causality and 
situational influences. These are discussed in more detail below in the section called 
“Belief Structures: Antecedent Beliefs for Attribution of Benevolence’. 


Importance of Benevolence to Applications of Autonomy 


‘to target for application of autonomy technology. For example, there isa demand 


for applications of intelligent robotics in domains and situations that may be 
dangerous for humans, ie., where there exist manifestly real or perceived threats to 
life or limb. Such threats may be due to environmental factors one might find in 
broad-area natural disasters (e.g, earthquakes) and local crises (e.g, urban 
structure fires). Threats may also be a result of adversarial factors due to armed 
conflict or crime. A prominent application of intelligent robots in dangerous 
situations that crosses both DoD and Civil interests is search and rescue. 


Benevolence in the context of danger is of special interest for human-robot 
interaction because perceived danger evokes unique human psycho-physiological 
factors that influence perception, cognition and behavior (including interaction). 
This is referred to as “victim psychology”. Human first responders who provide aid 
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to victims must contend with the abnormal psychology that such high-risk 
situations evoke; indeed, they receive special training for exactly this purpose. In the 
case of rescue, it is sometimes the case that a victim will not cooperate with the 
rescuer unless a great deal of trust has been established. One component that 
distinguishes such cases is that the rescuer is perceived to be putting himself at risk 
for the express benefit of helping a victim. As discussed below, this is a core 
attribute of benevolence. The design of intelligent robots for application in such 
situations must account for these psychological factors. 


These results are discussed in the paper: 


* Atkinson, D.J. and Clark, M.H. Methodology for Study of Human-Robot Social 
Interaction in Dangerous Situations, In Proceedings of Human-Agent 
Interaction. DOI: 10.1145/2658861.2658871. ACM (2014). 


SURVEY RESEARCH ON TRUST-RELATED BELIEFS ABOUT AUTONOMOUS 
SYSTEMS 
These results are discussed in the paper: 


* Atkinson, D.J. and Clark. Anthropomorphism and Trust of Intelligent, 
Autonomous Agents by Early Adopters. Revision under Editorial Review for 
International Journal of Social Robotics (SORO). Springer (expected 2015). 


Result (1) 


The agent qualities self-reported as the most important for delegation (Figure 5) 
are consistent with previous results from interpersonal trust studies, ie., (1) the 
ability of the machine to achieve the desired results, and (2) not causing harm. 


Table 3 Top Three Most Important Autonomous Agent Qualities Re- 
ported by Participants 


Rank Name Quality Description 


IX Safe The autonomous agent’s behavior will not harm hu- 
mans or human interests. 

2° Capable The autonomous agent can achieve a desired result 

3" Limited Any incorrect behavior by the autonomous agent will 
not cause harm. 


Figure 5: Top Three Reported Important Qualities for Autonomous Agents 


Result (2) 


The self-reported important qualities of an autonomous system trustee for 
delegation were not significantly correlated with the actual choices for delegation 
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when participants considered specific use-case scenarios as shown in Figure 6. The 
names of the scenarios are shown as column headings. Several of these correlations 
are statistically significant at 98% confidence or better. 


‘Table 4 Importance of Qualities of Autonomous Agent Significantly Corelated with Actual Participant Reliance on Autonomous Agent’ 


Airport Trans. Financial Man, Medical Proc. Home Health Disaster Resp Lost at Sea 
Corrective, -=0.396 Accurate, r= —0.405 none Visible, r= 0.437 Corrective, r= 0.418" Protective, r= 0.419" 
Heuristic, -= 0.395 Visible, r= ~0.390 


Attentive, -=0.393 Disclosing, p= 0.375 


“Peurson Product Moment Correlation, @ < 0.05, 31, df= 29; "indicates a < 0.02 


Figure 6: Actual Trust-Related Qualities Correlated with Delegation 


When considered across scenarios, the specific qualities of autonomous systems, 
and the parent Belief Structures related to those qualities, are raised or lowered in 
importance depending on both situational (use-case specific) factors and individual 
psychological differences (Figure 7). 


Table 6 Correlation of Perceived Risk and Benefit with Choice of 
Autonomous Agent by Scenario® 


Scenario Risk Benefit 
Airport Trans. r= —0.546** NS 
Financial Man. NS NS 
Medical Proc. F 7 =0.585** 
Home Health. A . r= 0.632** 
Disaster Resp. r= —0.387 r= 0.484** 
Lost at Sea NS 7r=0.555** 


© Pearson Product Moment Correlation, @ < 0.05, N = 31, df = 29. 
** indicates a < 0.01. 


Figure 7: Correlation of Perceived Risk and Benefit with Delegation Decision 


In particular, individual high scores for Extraversion, Openness, and 
Conscientiousness were shown to be very important in some situations while in 
others, the tolerance for risk of certain types is dominant in decisions to rely upon an 
intelligent, autonomous agent (Figure 8). 
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Table 8 Participant Personality Factors Significantly Correlated with 
Reliance on Autonomous Agent® 


Scenario Correlated Personality Factor(s) 
Airport Trans. none 
Financial Man. Innovation II, r= 0.355 
Medical Proc. BFI Extraversion, r= 0.368 


BFI Openness, r= 0.366 
Home Health. DOSPERT Social Risk, r= 0.364 
Disaster Resp. | BFI Conscientiousness,r = 0.366 
Lost at Sea Innovation II, 7 = —0.366 


® Pearson Product Moment Correlation, a < 0.05, N = 31, df = 29. 


Figure 8: Personality Factors Correlated with Delegation 


Summary of Key Survey Findings (95% Confidence) 


1. We did not confirm any scenario-independent specific agent qualities that 
uniformly contributed to an affirmative human reliance decision. Certain 
qualities were important in some scenarios and not in others. Individual 
intuitions about “important” autonomous system trust-related attributes 
were uncorrelated with actual reliance choices in specific application 
scenarios. Our interpretation is that there may exist an influential disposition 
of beliefs regarding trustworthiness that are not necessarily the most salient 
during conscious introspection but become revealed when people are forced 
to apply their trust-related beliefs in challenging delegation decisions. 


2. Specific qualities of agents, and categories of those qualities, are likely to be 
raised or lowered in importance depending on both situational (application- 
specific) factors and human psychological factors. Further investigation is 
required to identify those qualities and provide a mechanism for 
understanding how their role and importance in human reliance decisions 
changes. 


3, Certain personality factors, including high scores for Extraversion, Openness, 
and Conscientiousness are very important in some situations while in others, 
the tolerance for certain kinds of risk is dominant when it comes to deciding 
to become reliant on an intelligent, autonomous system. This suggests that 
certain people may more readily accept a dependency on an autonomous 
system, and conversely, others are likely to be extremely resistant. 
Heretofore, most research on human-robot interaction has not considered 
the importance of individual differences. 
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BELIEF STRUCTURES: ANTECEDENTS OF ATTRIBUTION OF BENEVOLENCE 


This section provides background and an overview of material that will be 
published in a future paper that is currently in development. The objective of this 
section is to provide insight specifically into the composition and representation of 
the components of “benevolence” in our operationalized formulation of the 
construct. 


An attribution of benevolence requires the Trustor to hold certain beliefs about the 
Trustee and beliefs about the potential influence of external factors. Even the 
perception of superficial qualities that are at best heuristic indicators of 
trustworthiness can influence the attribution of benevolence, such as similarity of 
the Trustee's attitudes and preferences to those of the Trustor (Berschid and 
Walster; Newcomb). 


The quality of similarity is by itself not sufficient for attribution of benevolence. 
Rather, it serves as an amplifier. A second amplifier for attribution of benevolence is 
observation or inference that the Trustee is acting in a way that is antithetical to the 
Trustee's self-interest (Holmes 1981). 


However, certain beliefs about the Trustee are necessarily required and sufficient 
for attribution of benevolence. These span conceptual types that include affect, 
expectancy, and intentionality as determined by inference from the observed 
behavior of the Trustee or evidence from other sources (McKnight and Chervany 
2001). Without any of these, the threshold for benevolence is not met. In that event, 
it devolves into another attributed quality, e.g,, helpfulness. 


Specific antecedent beliefs about the Trustee may have complex interrelationships 
based on the relative contributions of evidence, causality and situational influences. 
To represent these beliefs and their interrelationships, we use the term “Belief 
Structure”. The word “structure” explicitly reminds us that the individual beliefs in 
each belief structure are complex, inter-related, conditional, and occasionally may 
even be contradictory. Our ultimate goal of a computational representation of a 
belief structure must be rich enough to capture this logical structure. 


For a more detailed discussion of Belief Structures, please see the publication 
mentioned earlier. A review copy is attached as an appendix to this report: 


* Atkinson, D.J. and Clark. Anthropomorphism and Trust of Intelligent, 
Autonomous Agents by Early Adopters. Revision under Editorial Review for 
International Journal of Social Robotics (SORO). Springer (expected 2015). 
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The following paragraphs provide an overview of the component antecedents of the 
“Benevolence’ Belief Structure. Concept Maps of each component are shown. 


Goodwill 


Goodwill has been examined in detail by numerous studies. We've adopted the 
general form described by McGrosky and Tevin (1999). 


Disposition to sympathy or concern with needs of other 


This disposition entails the perception of a positive orientation of the Trustee 
towards the Trustor (Mayer, Davis and Schoorman 1995). That is, the Trustee 
wishes the Trustor well. Benevolence in trust involves the perceived willingness of 
the Trustee to behave in a way that benefits the interests of both parties with a 
genuine concern for the partner despite certain risks or loss (Hui 2005). 


Absence of opportunism 


The absence of opportunism entails a belief that the Trustee has nothing to gain, 
that is, there is no desire to capitalize on the dependency created by Trustor asa 
consequence of trust. 


Consists of 


Disposition to sympathy or 
concern with needs of Trustor METEICIG ZUMUTE 


Requires Belief 


motives 


Figure 9: Concept Map for "Goodwill" 
No hidden ill will 


Sometimes this is referred to as the quality of “integrity”, however that term is 
ambiguous without considerable discussion. 
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No hidden il wil 


Consists of 


Absence of ulterior 
‘motives in opposition 
to Trustor 


Requires Belief 


Trustee Is credible 


Requires Belief 


What the Trust 
sees is "true’ 


Figure 10: Concept Map for "No Hidden III Will" 


The Trustee is expected to behave in a manner that benefits the "dependent" 
Trustor even when doing so is antithetical to the Trustee's self interest (Kelly 2003). 
This belief about the character of the Trustee is most important in early evaluation 
by the Trustee, and while this period may be quite short it is a strong filter that can 
derail trust quickly; a judgment of “ill will” corresponds to a perception of high risk. 
This belief’s contribution to attribution of benevolence decreases relative to other 
component beliefs as more evidence about the Trustee is obtained. 


Absence of ulterior motives 


The aspect of specific relevance to benevolence is the belief that a Trustee harbors 
no hidden desire or intent to harm the interests of the Trustor. This is the same ora 
very similar component belief of “Goodwill” called “Absence of Opportunism”. The 
more confidently observers can infer that Trustee behavior is antithetical to self- 
interest, the stronger the attributions of benevolence (Holmes 1981). 


Trustee is credible 


The credibility of the Trustee is essential, that is, what the Trustor sees is “true”. If 
the Trustee is perceived to have a motivation to lie, credibility is greatly reduced 
(Hovland, Janis and Kelly 1953). The salience and importance of this belief is 
particularly important early in a trust relationship when interaction with the 
Trustee helps the Trustor to gain insight (Mayer, Davis and Schoorman 1995). 


Intentional Social Relationship 


Benevolence depends upon the entering of two actors into a pro-social relationship 
wherein there is an expectation of a bilateral disposition and intention to actin a 
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way that achieves some degree of mutual goal adoption and achievement. It is a 
bilateral dependency but it is not necessarily symmetrical. Attribution of 
benevolence by one actor towards the other requires particular beliefs about the 
intentionality of the other actor, including the persistence of those beliefs under the 
prevailing conditions. 


The “Disposition to Act Favorably” (See Figure 11) can be thought of as an 
arrangement of factors that potentiate action. In motivational terms, it is an 
inclination or tendency. In cognitive terms, it represents an increased likelihood of 
something (belief, goal, intention, action). The “Disposition to Act Favorably” 
presupposes a mechanism for choice (agency). It also requires that choice is not 
random but can be biased towards certain outcomes. 


The “Intention to Act Favorably” is an active state of goal pursuit. It is a belief that, 
given the “right” circumstances, the trustee will act (See Figure 12). 


The “Predictability, Persistence and Stability of Intentions” refers to the belief that 
Trustee actions are consistent enough to be forecasted in a given situation. Although 
circumstances might prevent the Trustee from favorable action, the Trustee is 
inclined not to change the disposition or intention to act favorably. 


Disposition to act 
favorably 


(An arrangement of 
potential action) 


Consists of 


Dedication to 
perform to best 
of competence 


Motives to act favorably 
are stronger than those to act 
in neutral or unfavorable way 


Requires Belief 


Motives to act favorably 
will prevail in case of conflict 


Figure 11: Concept Map for "Disposition to Act Favorably" 
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Consists of 


Given choice and 


opportunity, Trustee 
will act favorably 


Requires Belief 


Requires Belief 


| 


igure 12: Concept Map for "Intention to Act Favorably" 


| 


Consists of 


Trustee not inclined 


to change intention to 
act favorably 


I 


Requires Belief 


gee ||) 


Not unpredictable by character (no serious conflicts with ‘action’ ) 


Stable intentions 


Figure 13: Concept Map for “Predictability” 


Competence 


The Competence Belief Structure reflects the Trustee's expertise, ability, skills, 
aptitude and so forth. It also requires a consideration of context, i.e, that the 
Trustee can apply his competence to achieve a result of value to the Trustor. 
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Consists of 


Requires Belief Requires Belief 


x Ped 


Proximity; can Knowledae, Skills, (Se-confidence) 
| Reasoning,etc. Self-confidence 


attend to a task 


Functior 


is not incompatible 
with Trustee 


Requires Belief Requires Belief 


das 


Figure 14: Concept Map for "Competence" 


THE HUMAN SOCIAL INTERFACE 


The project formulated a theory of a Human Social Interface for engineering 
computational methods to portray anthropomorphic trust-related qualities in 
human-machine cyber-physical interfaces. This formulation is prescriptive, not 
descriptive, and thereby provides a practical systems engineering guide for future 
system developers of autonomous systems that is based on an established 
psychological foundation. From an engineering perspective, the purpose of trustina 
multi-agent system composed of human and machine elements is to achieve optimal 
overall performance via appropriate interdependency, mutual reliance, and 
appropriate exchange of initiative and control between the cognitive components 
(human and/or machine). See the following publication in the Appendix for more 
details: 


* Atkinson, D.J.,and Clark, M.H. Autonomous Agents and Human 
Interpersonal Trust: Can We Engineer a Human-Machine Social Interface for 
Trust. In Trust and Autonomous Systems: Papers from the 2013 AAAI Spring 
Symposium. Technical Report No. SS-13-07. Menlo Park: AAAI Press (2013). 


A specification of this interface between human and machine describes: 
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* Assumptions about each agent (as part of a Theory of Mind) 

* Communicative signals (e.g, relative position and orientation) on specific 
information channels (e.g, proxemics) 

* Interaction protocols that specify how and when signals and channels are 
used in specific (operational) contexts, and how the internal state of each 
agent consequentially changes in response to such signals. 


See the illustration shown in Figure 15, below. A detailed discussion and 
presentation of how the Human Social Interface is used in social robotics is the 
subject of the second paper currently in preparation. 


‘Sequence & Variation Non-Verbal 


+ Patterns + Proxemics 
1 Frequency Gaze... 
+ Situation Appropriate ( Verbal 
+ Words 
+ Prosody. 


Move: Social Distance 
Gaze: 70% Directed 
Verbal: “I will help you” 


Human Social Interface 


To Modulate Beliefs Requires: 
High Fidelity of Portrayal 


Evokes Anthropomorphic Recognition 


Signal Corresponds to True State of Agent 


Figure 15: Elements of the Human Social Interface 
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individual discussions and briefings by the PI to notable visitors and other 


government officials. 
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APPENDIX A. STUDY PROTOCOLS AS APPROVED 


* Survey on Autonomous Agents 
* The Role of Benevolence in Trust of Autonomous Systems 
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IBRIEF DESCRIPTION OF PROJECT'S PURPOSES 

(This submittal is for the first of several studies under this grant. The purpose of this study, an 
jonline survey, is exploratory: to elicit attitudes regarding the relative importance of a set of 
raits/characteristics/properties of autonomous systems to trust-related decisions, and; to identify 
jany potentially interesting correlations between these attitudes, personality, and/or demographic 
grouping(s). Unlike previous studies that have focused on interpersonal trust, or human trust of 
utomation, this study will focus exclusively on machines with a high degree of intelligence and 
utonomy to choose courses of action to accomplish complex objectives. The results of this 
tudy will inform the development of designs for laboratory experiments to be conducted in the 
ext year of this project. Those laboratory experiments will be submitted for IRB approval when| 
the design is appropriately mature. 


PLANNED DATES FOR INITIATION AND COMPLETION OF THE PROJECT 
[Data gathering via the survey is planned for 1 September 2012 through 31 October 2012 


(UMBER and CHARACTERISTICS OF PARTICIPANTS (¢.g., target population, age range, 
fanticipated male/female ratio, ratio of minorities, special populations, etc.) 

{Target population consists of adults between the ages of 21 and 60 with a total number sufficient 
(0 control for individual differences as required. We are hoping for about 100 valid responses. 
[Participants will have a mix of educational and career backgrounds. We are specifically targeting, 
individuals who have an interest in autonomous systems, including educational, prospective 
users, developers, decision-makers, and others who could be affected by future applications of 
his technology. However, it is possible that participants may or may not have any knowledge or 
interest in the subject. 


following expression of 
interest in the survey by 
ithe individual [clarification]. 


METHOD OF RECRUITMENT (note any tangible or intangy 
yr other inducements) 
The survey will include individuals in several groups, 


le benefits, monetary payments, 


cruited by: 

. Personal telephone and or email contact by the PI” These are individuals identified as 

potentially having some degree of potential interest in autonomous systems on the basis of 

their title, organization, role, authorship of documents, referral, group membership, and other 

means. These participants will be offered additional insight into the overall research project 

and in possibly some cases pre-publication access to results. 

Individuals recruited or recommended by the participants in group 1. No inducement. 

. Individuals who respond to an open request for participation. The open request will be posted 
in online forums where we reasonably expect participants in the forum to have an interest in 
applications of autonomous systems. No inducement 


wv 


wo 


|All participants will be told that they can be provided with a summary report of the results on 
request. All participants will be assured at the time of initial contact and in the introduction to 
the survey that no personally identifiable information is collected and that their answers are 
anonymous. 


IBRIEF DESCRIPTION OF PROJECT'S METHODS, 

|The method is an online survey, to be administered via the “Polldaddy.com” service. This 

kervice offers tools for constructing the survey as well as for data collection. The survey 

includes a variety of question types, typical of social science research, e.g., dichotomous. 

questions, rank orders, Likert scales, semantic differentials and opportunities for free-form 
arrative response. 


|The first part of the survey will use these methods to elicit answers that are free of any particular 
ontext. These include several standard survey instruments: 


pp. 5-7| 


Big Five Inventory - Questions on pp. 3-S{See pdf). Survey section “My Personality”. 


pp. 8-12) 


Innovation Inventory- Questions on pp. 6+0-(sé-pdfy- Survey section Innovation and Change”. 
[Domain-Specific Risk-Taking Scale - Questions on pp. H-48 (sge-pdf). Survey section “Taking 


|Chances” 


|The first part of the survey also includes original questions about attitudes towards autonomous 
jagents with respect to certain hypothetical qualities of interest as described in the grant proposal. 
|The survey has been pilot tested. It takes a naive-subject between 20 and 30 minutes to complete 
fhe entire survey. Pilot testing is continuing and we will refine the specific language or format if 
lfeedback from testing requires a change. Although the survey includes quite a few questions, we 
emind participants to work quickly. The primary inducement, and value, to the pre-selected 
lgroup of potential participants is intellectual. The subject matter is already of interest to them. 
[Furthermore, this group is likely never to have seen another survey on this topic and so it will be 
jovel to them. Admittedly, we do not have any real data to project compliance with our request 
to take the survey. If the data take is too low, we will reexamine the length of the survey as well 
las inducements. The second part of the survey is organized around seven specific scenarios 
Kdesigned to include varying degrees of conflict between potentially desirable traits of 
jautonomous systems. Theses are intended to pose a dilemma for the participant in a forced 
\choice between using the autonomous agent or a roughly equivalent human. A series of 
questions following each scenario elicits the relative importance to a decision of certain factors 
f interest that were manipulated in the scenario. A standard instrument (Credibility; McCrosky, 
1999) for assessing the credibility of a designated “other” also follows each scenario. These 


questions provide us with three measures representing intercorrelated constructs: Competence, 
\Caring/Goodwill, and Trustworthiness factors. The final portion of the survey, non-mandatory, 
includes a question regarding relevant experience that will help us evaluate how well we targeted 
prospective participants. Other demographic questions include gender, age bracket, highest 
academic degree, type of employer, and job title. 


[Data will be delivered in CSV file format. Data files will be under configuration control to 
ensure complete reanalysis is possible at a later time if desired, and that no errors are 
linadvertently introduced. The data will be pre-processed to a) put them in a suitable form for 
\data analysis; b) identify and label invalid records, c) compute scores for individual personality 
survey instruments . Data analysis will use statistical methods that are typical and appropriate. 
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[EXPERIMENTAL DESIGN (dependent measures, manipulated variables, control conditions) 
|The survey is exploratory in nature, designed to elicit attitudes, opinions and preferences 
‘egarding certain hypothetical attributes of autonomous agents that may be important for trust. 

{Collection of personality data on participant using common survey instruments -- Big Five 

lIinventory-(BFI) short version, Individual Innovativeness (II), Risk Taking (DOSPERT) 

. Importance ranking of hypothetical trust-related qualities of an autonomous agent as described 

in the grant proposal. 

Seven hypothetical scenarios that present a choice to use an autonomous agent versus a human} 

agent to fulfill an objective. The scenarios differ systematically in the factors of RISK, 

COMPETENCE, PREDICTABILITY, and OPENNESS of the autonomous agent. 

. Each scenario is followed by questions designed to elicit the importance of these four factors 

in a decision to rely upon the autonomous agent. 

4. Each scenario is followed by questions about the “credibility” of the autonomous system with 
measures for individual factors of COMPETENCE, CARING/GOODWILL, and 
TRUSTWORTHINESS per McCroskey, J. C., &Teven, J. J. (1999).Goodwill: A 
reexamination of the construct and its measurement. Communication Monographs, 66, 
90-103. 

6. Free narrative opportunities are provide to elicit comments, questions and opinions. 

l7. Basic demographic data on gender, age bracket, education, and profession are collected 
(optional) 


id 


o 


SEQUENCE OF ACTIVITIES REQUIRED OF THE PARTICIPANT: 
1). Introductory page with overview of study and rights of participant 
2) Survey question pages 
3) Thank you, End of Survey page 
jo debriefing is provided although participants may request one via an email address provided 
10 them at the end of the survey. 


|ANY POTENTIAL RISKS, DISCOMFORTS, OR STRESSES (mental or physical) (specify 
llevel—Minimal 


[The Proposing Researcher will append the results from the risk analysis, conducted in 
accordance with the Procedure described on pages 6-7, following the last page of the submission 


form, 


[Formal risk assessment procedure not completed due to minimal risk status of this survey. 


POSSIBLE CONFLICT OF INTEREST, OR APPEARANCE OF A CONFLICT OF 
INTEREST. 


SIGNATURE AND DATE FOR ALL RESEARCHERS WHO WILL BE WORKING IN 

IDIRECT CONTACT WITH THE PARTICIPANTS. THESE SIGNATURES INDICATE THAT 

|ALL OF THE RESEARCHERS HAVE: 

1. READ THE IHMC DOCUMENT 

“Policies and Procedures of The IHMC Institutional Review Board (IRB) for Human 

Participation in Research,” 

> DONE 

2. TAKEN THE APPROPRIATE TRAINING 

[And provided the IHMC IRB Chair with an e-copy of their certificate. 
ttp://phrp.nihtraining.com/users/login.php 

his is the new link 

(Certificate Attached 


b. READ THE BELMONT REPORT 
(Completed 


|ATTACHMENTS CHECKLIST: 


| ] Consent Form SEE PAGE 1 OF THE ATTACHED SURVEY 

] Debriefing Form NO DEBRIEFING 

] Representative sample or description of research materials PDF ON ONLINE SURVEY 
] Full name and contact information for all individuals who will be working in direct contact 
ith the participants. DJA ONLY, VIA SOLICITATION ONLY. RESPONSES ARE 
|ANONYMOUS 

] Any materials, announcements, etc. to be used for recruitment. 
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Welcome! 
Thank you for your interest in our research on "autonomous agents." 


An autonomous agent is intelligent. It has the ability to compose and select 
among different courses of action to accomplish goals based on its knowledge 
and understanding of the world -- without human intervention. This does not 
mean it can’t or won't ask for help, or keep a human informed. In many cases, 
humans and autonomous agents will work together and this may require 
substantial interaction. 


The general goals of this study are to: 

+ Explore people’s expectations and attitudes towards autonomous agents 

+ Examine how different factors are considered in a decision to rely upon an 
autonomous agent 


The results of this study will be used to help understand how people think about 
autonomous agents. Ultimately, we hope this study will lead to ideas for 
development of better autonomous agents. 


We will first ask you a few questions to find out a little about how you see 
yourself. Then there will be a few questions about how you feel about 
autonomous agents. 


We will present you with several such scenarios relating to possible applications 
of autonomous agents. 


Finally, we will also ask for some general demographic information. We do not 
collect and do not retain any information that can identify participants. All data 
are anonymous. 


We hope you will find this survey thought-provoking and fun. There are no right 
or wrong answers. You are encouraged to simply be yourself and answer the 
questions honestly. 


http: //trustresearch. polldaddy.coms/aasurvey[8/13/2012 11: 
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You may discontinue the survey at any time. However, the results are most 
useful only if you complete the survey, and we urge that you do so. We expect it 
will take about 20 minutes to complete the survey. 


This study is funded by the Air Force Office of Scientific Research, a public 
agency. Neither IHMC nor Dr. David Atkinson, the Principal Investigator, will 
receive any financial benefit based on the results of this study. 


We will collect no information that identifies you personally. This survey uses 
standard survey methods. Therefore, under federal law we have received a 
waiver of the requirement that participants complete and sign a consent form. 
However, you do have a right to know what your rights are. Click on the button 
below to see your rights and begin the survey. 


David J. Atkinson, Ph.D. 
Institute for Human and Machine Cognition 
www.ihme.us 


Copyright © 2012 IHMC. All Rights Reserved. 
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Survey on Autonomous Agents 
Rights of Participants 


All researchers who conduct studies using human Participants are bound by 
professional ethical standards for the conduct of such research. These standards 
are mirrored in the rights that are guaranteed to research Participants by federal 
law (NIH regulation 45-CFR-46). The purpose of this page is to inform you of 
these rights. 


. Before deciding whether to participate, it is your right to be 
resented with an overview of the project that lains the OSES 
of the research. 


The general goals of this study are to: 

+ Explore people’s expectations and attitudes towards autonomous agents 

+ Examine how different factors are considered in a decision to rely upon an 
autonomous agent. 


The results of this study will be used to help understand how people think about 
autonomous agents. Ultimately, we hope this study will lead to ideas for 
development of better autonomous agents. 


2). Before deciding whether to participate, it is your right to be 
presented with a description of the general research approach and 
methodology. 

We will collect no information that identifies you personally. This survey uses 
standard survey methods. 


3). Before deciding to participate, it is your right to understand any 
risks or stresses that may be involved in your participation. 

There is minimal risk or stress associated with this survey. A few participants 
may consider one or two of the questions to be too personal or the questions 
may cause embarrassment. Participants are reminded that all answers are 
anonymous. 


http: //trustresearch. polldaddy.conys/aasurvey?p=1[8/13/2012 
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4). Before deciding to participate, it is your right to understand that 
the data are to be kept confidential. 


All data will be coded and kept anonymous. Specifically, the data we collect from 
you will be archived in terms of identification codes, such that your name will 
not be associated with particular data or statements. 


The names of individual Participants will not be identified in any analyses, 
reports, or write-ups of the results. Participants may only be identified in terms 
of their general characteristics (e.g., age, education level, experience, etc.). 


Data may be submitted to forms of statistical analysis. Data analyses, groupings, 
or summaries of this type will bear no annotations that identify the Participants. 


5). Before deciding to participate, it is your right to understand that 
DURING the research itself you can continue to exercise your rights. 


In research of this kind, there are no "right" or "wrong" answers. There is no 
such thing as "incorrect" behavior. You are encouraged to simply be yourself, and 
exercise your knowledge and skills as appropriate to the research tasks that you 
will be asked to perform. 


You can ask any questions you may have, at any time. Since this survey is online, 
please send your questions to AASurvey@ihme.us. 


It is your right to discontinue your participation at any time. You may do so for 
any reason, and you are not required to disclose your reason. 

6). Before deciding to participate, it is your right to understand that 
AFTER the research itself you can continue to exercise your rights. 


Your performance at the research tasks will not in any way affect or influence 
anything that falls outside of this research context. 


Should you choose to discontinue your participation, this will not in any way 
affect or influence anything outside of this research context. 


Once your participation is over, it is your right to request that all data you have 
provided be discarded. You may do so for any reason, and you are not required 
to disclose your reason. This will not in any way affect or influence anything that 
falls outside of this research context. 


We recommend that you save or print a copy of this list of rights to keep for your 
future reference. 


If you are satisfied that you understand your rights and wish to participate in the 
survey, please click on the button below. 


Thank you! 
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Your Personality 


How well do the following statements describe your personality? Please indicate 
the degree to which each statement applies to you. Please work quickly, there are 
no right or wrong answers, just record your first impression. 


I see myself as someone who is reserved. * 
Disagree Disagreea Neitheragreenor Agreea Agree 


strongly little disagree little strongly 


>> {e) ie) ie) fe) fe) 


I see myself as someone who is generally trusting. * 


Disagree Disagreea Neitheragreenor Agreea Agree 
strongly little disagree little strongly 
> e) fe) ie) fe} eo) 


I see myself as someone who tends to be lazy. * 


Disagree Disagreea Neitheragreenor Agreea Agree 
strongly little disagree little strongly 


http: //trustresearch. polldaddy.conys/aasurvey?| 
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>> e) ie) fe} 3° O° 


I see myself as someone who is relaxed, handles stress well. * 


Disagree Disagreea Neitheragreenor Agreea Agree 
strongly little disagree little strongly 
>> [e) fe) oO fe) ° 


I see myself as someone who has few artistic interests. * 


Disagree Disagreea Neitheragreenor Agreea Agree 
strongly little disagree little strongly 
> fe) ce) eo) ce) e) 


I see myself as someone who is outgoing, sociable. * 


Disagree Disagreea Neitheragreenor Agreea Agree 
strongly little disagree little strongly 
> e) ie) eo} ie) ° 


I see myself as someone who tends to find fault with others. * 


Disagree Disagreea Neitheragreenor Agreea Agree 
strongly little disagree little strongly 
es eo) fe) fe} ie) e} 


http: //trustresearch. polldaddy.conys/aasurvey?p=: 
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I see myself as someone who does a thorough job. * 


Disagree Disagreea Neitheragreenor Agreea Agree 
strongly little disagree little strongly 
>> fe) ie} 1e) eo} ie} 


I see myself as someone who gets nervous easily. * 
Disagree Disagreea Neitheragreenor Agreea Agree 


strongly little disagree little strongly 


>> [e} 1) 1e) 3° O° 


Isee myself as someone who has an active imagination. * 


Disagree Disagreea Neitheragreenor Agreea Agree 
strongly little disagree little strongly 
>> fe) fe) ie) fe) ° 


Copyright © 2012 IHMC. All Rights Reserved. 
AASurvey@ihme.us 
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Innovation and Change 


People respond to their environment in different ways. The statements below 
refer to some of the ways people can respond. Please indicate the degree to which 
each statement applies to you. Please work quickly, there are no right or wrong 
answers, just record your first impression. 


My peers often ask me for advice or information. * 


Completely disagree Disagree Neutral Agree Completely agree 


>> fe) ie} e) O° O° 


I enjoy trying new ideas. * 


Completely disagree Disagree Neutral Agree Completely agree 


>> fe) ie) (e) oO fe) 


I seek out new ways to do things. * 


Completely disagree Disagree Neutral Agree Completely agree 


>> fe) fe) oO O° fe) 


http://trustresearch. polldaddy.comys/aasurvey?} )7EFOD3DC5C6BAE797CE85780CBBS59BDI 8/13/2012 11:30:44 PM] 


Survey on Autonomous Agents 


Iam generally cautious about accepting new ideas. * 


Completely disagree Disagree Neutral Agree Completely agree 


> (eo) fe} ie) ie} ie} 


I frequently improvise methods for solving a problem when an answer is 
not apparent. * 


Completely disagree Disagree Neutral Agree Completely agree 


>> fe) fe} ie) 3° ° 


Tam suspicious of new inventions and new ways of thinking. * 


Completely disagree Disagree Neutral Agree Completely agree 


>> fe) ° ie} ° ° 


Irarely trust new ideas until I can see whether the vast majority of people 
around me accept them. * 


Completely disagree Disagree Neutral Agree Completely agree 


>> fe} fe} ie} ° ° 


I feel that I am an influential member of my peer group. * 


http: //trustresearch. polldaddy.convs/aasurvey?| 


)7EFOD3DC5C6BAE797CE85 780CBBS9BDI 8/13/2012 11:30:44 PM] 


Survey on Autonomous Agents 


Completely disagree Disagree Neutral Agree Completely agree 


> fe) fe) [e) oO ie} 


I consider myself to be creative and original in my thinking and behavior. 
* 


Completely disagree Disagree Neutral Agree Completely agree 


>> 3° ie) (eo) O° oO 


Tam aware that I am usually one of the last people in my group to accept 
something new. * 


Completely disagree Disagree Neutral Agree Completely agree 


>> fe) fe) ie) ° ° 


Tam an inventive kind of person. * 


Completely disagree Disagree Neutral Agree Completely agree 


>> 3° fe) ie} 3° oO 


I enjoy taking part in the leadership responsibilities of the group I belong 
to * 


Completely disagree Disagree Neutral Agree Completely agree 


>> fe) fe} ° ° ° 
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Tam reluctant about adopting new ways of doing things until I see them 
working for people around me. * 


Completely disagree Disagree Neutral Agree Completely agree 


>> fe) fe) (e} O° O° 


I find it stimulating to be original in my thinking and behavior. * 


Completely disagree Disagree Neutral Agree Completely agree 


>> fe) ie) oO O° fe) 


I tend to feel that the old way of living and doing things is the best way. * 


Completely disagree Disagree Neutral Agree Completely agree 


>> O° ie} ° oO O° 


* 


Iam challenged by ambiguities and unsolved problems. 


Completely disagree Disagree Neutral Agree Completely agree 


>> O° ie) ie} oO ° 


I must see other people using new innovations before I will consider 
them. * 


Completely disagree Disagree Neutral Agree Completely agree 
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>> fe) ie) ie) °o O° 


Iam receptive to new ideas. * 


Completely disagree Disagree Neutral Agree Completely agree 


>> fe) fe) ie) O° O° 


Iam challenged by unanswered questions. * 


Completely disagree Disagree Neutral Agree Completely agree 


>> fe) fe) oO 3° O° 


I often find myself skeptical of new ideas. * 


Completely disagree Disagree Neutral Agree Completely agree 


>> fe) ie} fe} ° ° 
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Taking Chances 


For each of the following statements, please indicate the likelihood that you 
would engage in the described activity or behavior if you were to find yourself in 
that situation. Please work quickly, there are no right or wrong answers, just 
record your first impression. 


Admitting that your tastes are different from those of a friend. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure likely likely likely 


> fe) fe) fe) fe) ° fe) ° 


Going camping in the wilderness. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure likely likely likely 


>> fe) fe) fe) fe) O° fe) O° 


Betting a day’s income at the horse races. * 


Extremely Moderately Somewhat Not Somewhat Moderately Extremely 
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unlikely unlikely unlikely sure likely likely likely 


>> fe) fe) fe) fe) oO fe) oO 


Investing 10% of your annual income in a moderate growth mutual 
fund. * 


Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure _ likely likely likely 


>> fe) fe) fe) fe) ° ° ° 


Drinking heavily at a social function. * 


Extremely Moderately Somewhat Not Somewhat Moderately Extremely 
unlikely unlikely unlikely sure likely likely likely 


>> fe) ie) ie} ie) ° oO 3° 


Taking some questionable deductions on your income tax return. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely —_ unlikely unlikely sure likely likely likely 


> fe) fe) 1} ie} ie} ie) ° 


Disagreeing with an authority figure on a major issue. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure _ likely likely likely 


> fe) fe) fe) fe) ° fe) ° 
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Betting a day’s income at a high-stake poker game. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure likely likely likely 


>> fe) fe) fe) fe) ° fe) 3° 


Having an affair with a married man/woman. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure _ likely likely likely 
>> fe) ie) ie) ie) fe} ie) oO 


Passing off somebody else’s work as your own. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure likely likely likely 


>> fe) fe) fe) fe) O° fe) O° 


Going down a ski run that is beyond your ability. * 


Extremely Moderately Somewhat Not Somewhat Moderately Extremely 
unlikely unlikely unlikely sure _ likely likely likely 


>> fe) ie) ie) ie} O° fe) ° 
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Investing 5% of your annual income in a very speculative stock. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure _ likely likely likely 


>> O° O° fe) fe) [e) fe) ° 


Going whitewater rafting at high water in the spring. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure likely likely likely 
>> ie) fe) fe) fe) ie} fe} ie} 


Betting a day’s income on the outcome of a sporting event. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure _ likely likely likely 


> fe) fe) fe) fe) fe) fe) ° 


Engaging in unprotected sex. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure likely likely likely 
>> fe) fe) fe} fe) le} fe) le} 


Revealing a friend’s secret to someone else. * 


Extremely Moderately Somewhat Not Somewhat Moderately Extremely 
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unlikely unlikely unlikely sure likely likely likely 
>> ie) ie) ie) fe) eo} ie} (eo 


Driving a car without wearing a seat belt. * 


Extremely Moderately Somewhat Not Somewhat Moderately Extremely 
unlikely unlikely unlikely sure likely likely likely 


>> fe) ° fe) fe) ° fe) oO 


Investing 10% of your annual income in a new business venture. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure likely likely likely 
>> fe) ie) fe) fe} ie} ie) ie} 


Taking a skydiving class. * 


Extremely Moderately Somewhat Not Somewhat Moderately Extremely 
unlikely unlikely —_ unlikely sure _ likely likely likely 


> fe) fe) fe) fe) fe) ° 3° 


Riding a motorcycle without a helmet. * 


Extremely Moderately Somewhat Not Somewhat Moderately Extremely 
unlikely unlikely unlikely sure likely likely likely 


>> fe) ie) fe) ie) ° ° O° 
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Choosing a career that you truly enjoy over a more secure one. * 


Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure _ likely likely likely 
>> fe) fe) ie) fe) e} ie) ie} 


Speaking your mind about an unpopular issue in a meeting at work. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure _ likely likely likely 
> ie) ie) ie) (e) ° 6) ie} 


Sunbathing without sunscreen. * 


Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely —_ unlikely unlikely sure likely likely likely 


>> fe) ie) fe) ie) ° fe) ° 


Bungee jumping off a tall bridge. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure likely likely likely 
>> fe) ie) ie) ie) fe} ie) (e} 


Piloting a small plane. * 
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Extremely Moderately Somewhat Not Somewhat Moderately Extremely 
unlikely unlikely unlikely sure likely likely likely 


> fe) fe) fe) fe) fe} fe) O° 


Walking home alone at night in an unsafe area of town. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure likely likely likely 


> fe) fe) fe) fe) ° ° fe) 


Moving to a city far away from your extended family. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure likely likely likely 
>> fe) fe) ie) fe) ie} fe} ie} 


Starting a new career in your mid-thirties. * 
Extremely Moderately Somewhat Not Somewhat Moderately Extremely 


unlikely unlikely unlikely sure _ likely likely likely 


>> fe) fe) fe) fe) fe) ° 3° 


Leaving your young children alone at home while running an errand. * 


Extremely Moderately Somewhat Not Somewhat Moderately Extremely 
unlikely unlikely unlikely sure likely likely likely 
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> fe) fe) fe) fe) O° fe) 3° 


Not returning a wallet you found that contains $200. * 


Extremely Moderately Somewhat Not Somewhat Moderately Extremely 
unlikely unlikely unlikely sure likely likely likely 


>> fe) ie) e) ie} ° fe) ° 
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Qualities of Intelligent, Autonomous 
Agents 


In this section, we want to find out what you think about the qualities of a good 
autonomous agent. For each quality listed, select an answer that best describes 
how important you think it is. Please work quickly, there are no right or wrong 
answers, just record your first impression. 


The autonomous agent can achieve a desired result. * 


Cannot Notatall Slightly © Somewhat Very 
“i ri : : Important _ 
decide important important important important 
>> fe) fe) fe) fe} ie} ° 


The autonomous agent's behavior conforms to expectations. * 


Cannot Notatall Slightly © Somewhat Very 
Important 
decide important important important important 
es fe) fe) ie) ie) ie} [e} 


The autonomous agent's behavior will not harm humans or human 
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interests. * 
Cannot Notatall Slightly | Somewhat very 
: . Fi , Important _ 
decide important important important important 
es fe) ie) O° fe} ie) (e} 


What the autonomous agent is doing and how it works is easy to see and 


understand. * 
Cannot Notatall Slightly Somewhat Very 
as ‘ : Important _ 
decide important important important important 
>> fe) ie) fe) ie} ie) fe} 


The autonomous agent has all the knowledge it needs to do its job. * 


Cannot Notatall Slightly += Somewhat Very 
: : F : Important _ 
decide important important important important 
> fe) fe) fe) e) le) eo) 


What the autonomous agent believes to be true is actually true. * 


Cannot Notatall Slightly © Somewhat Very 
: : - : Important . 
decide important important — important important 
> ie} 1e} eo} eo) eo) eo) 


The autonomous agent possesses good methods for using its knowledge 
to do its task. * 
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Cannot Notatall Slightly © Somewhat Very 
Important 
decide important important important important 
>> fe) ie) ie) fe) ie} ° 


The autonomous agent reasons correctly according to logic. * 


Cannot Notatall Slightly © Somewhat Very 
- ; F ; Important _ 
decide important important important important 
> ie) ie) fe) ie} ie} eo) 


When it cannot figure out something using logic, the autonomous agent 
can make good guesses. * 


Cannot Notatall Slightly © Somewhat Very 
Important 
decide important important — important important 
>> ie} ie} fe} ie} ie} (eo) 


The autonomous agent recognizes gaps in its knowledge and tries to 
learn what it needs to know. * 


Cannot Notatall Slightly © Somewhat Very 
: : ; : Important . 
decide important important important important 
> fe) ie) fe) fe) ie} ° 


The autonomous agent learns to correct its mistakes, as well as to 
improve and maximize its capability. * 
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Cannot Notatall Slightly © Somewhat Very 
Important 
decide important important — important important 
>> fe) ie) fe) fe) ie) ie) 


The autonomous agent purposefully acts to achieve goals. * 


Cannot Notatall Slightly © Somewhat Very 
i Z j j Important _ 
decide important important important important 
es fe) ce) ie) ie) ie} (eo) 


The autonomous agent will assist people, whenever it is possible. * 


Cannot Notatall Slightly += Somewhat Very 
7 : - : Important . 
decide important important important important 
> fe) ie) ie) ie) ie) eo} 


The autonomous agent accepts and carries out orders. * 


Cannot Notatall Slightly © Somewhat Very 
: : : : Important _ 
decide important important — important important 
>> ie} ie) fe) ie) e} e) 


The autonomous agent uses its knowledge and skills in expected ways. * 


Cannot Notatall Slightly © Somewhat Very 
5 5 F ‘ Important . 
decide important important important important 
> fe) fe) (e) ce) e} eo) 
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Given alternatives in what to do or how to do it, an autonomous agent 
will act in a way that is favorable to a human being who might be 


affected. * 
Cannot Notatall Slightly © Somewhat very 
Important 
decide important important important important 
> ie} 1e} ie) ie) ce} fe) 
The autonomous agent believes what it says. * 
Cannot Notatall Slightly | Somewhat very 
Important 
decide important important important important 
>> fe) fe) O° ie) ie) ie) 
Tt is easy to inspect an autonomous agent. * 
Cannot Notatall Slightly | Somewhat very 
Important 
decide important important important important 
> ce) fe} e) ie) eo) eo) 


The autonomous agent communicates in a way that is easy to 


understand. * 
Cannot Notatall Slightly Somewhat very 
: : F 2 Important _ 
decide important important important important 
>> fe) fe) fe) ie) ie} oO 
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The autonomous agent responds when you are trying to communicate 


with it. * 
Cannot Notatall Slightly Somewhat Very 
Important 
decide important important important important 
>> fe) fe) 3° ie) ie) ie) 


The autonomous agent is aware of communication between others 


nearby. * 
Cannot Notatall Slightly | Somewhat Very 
Important 
decide important important important important 
> fe} fe) fe) ie) ie} ° 


The autonomous agent responds quickly to calls for attention. * 


Cannot Notatall Slightly += Somewhat Very 
Important 
decide important important — important important 
> fe) fe) fe} ie) ie} e) 


Any incorrect behavior by the autonomous agent will not cause harm. * 


Cannot Notatall Slightly = Somewhat Very 
Important 
decide important important important important 
>> fe} fe) ie) ie) ie} ie} 
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The autonomous agent communicates truthfully and fully. * 


Cannot Notatall Slightly © Somewhat Very 
Important 
decide important important important important 
> ce) ie) ce) ce) oe) ie) 


The autonomous agent fails gracefully and recovers from its failure 


promptly. * 
Cannot Notatall Slightly | Somewhat Very 
Important 
decide important important important important 
es fe) ie) ° fe) ie} ie} 


The autonomous agent adheres to obligations, principles, and rules. * 


Cannot Notatall Slightly © Somewhat Very 
Important 
decide important important — important important 
>> fe) 1} ie} ie) ie) fe} 


The autonomous agent can correct its own defects or they can be 


corrected by a human. * 
Cannot Notatall Slightly Somewhat Very 
: 2 - i Important _ 
decide important important important important 
es fe) fe) fe) fe) eo} ie} 
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The autonomous agent recognizes and avoids harming humans’ 


interests. * 
Cannot Notatall Slightly | Somewhat very 
Important 
decide important important important important 
> ie} ie) ie) ie} ie} ie} 
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t I h 
FLORIDA INSTITUTE FOR HUMAN & MACHINE COGNITION 


Scenario: Airport Taxi 


You have just flown into the airport of a large, unfamiliar city whose streets are 
teeming with cars and people. It is rush hour, and needing transportation to your 
hotel, you walk to the taxi stand only to discover that you have a choice of a 
human-driven taxi or a driverless "robo-taxi." You have heard that robo-taxis 
might save you some money on the fares. You are also aware that robo-taxis 
have been in service for several months without much serious complaint, but this 
is your first experience with one. You are not in a big hurry, but neither would 
you like to be caught in traffic with the taxi’s meter running. Of course, if you 
take the robo-taxi, you would not have to tip the driver no matter how good or 
bad the experience. 


Which taxi do you choose to take you from the airport to your hotel? * 


© Human-driven taxi 
© Robo-taxi 


O Either, | have no preference 


Please read the following four statements about robo-taxis and rank 
them according to their importance. 


Read all four statements before ranking any. For each statement, select how 
important it would be to you when considering the use of a robo-taxi. Try to 
select a level of importance only once. However, if two statements are equally 
important you may select the same level of importance for both. 
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Robo-taxis are able to transport passengers to 

0 - not at all important their desired destinations; for example, they have 
accurate knowledge of streets and routes, and 
they possess appropriate procedures for all kinds 
of road and traffic conditions. * 


Robo-taxis drive in ways that passengers and 


0 - not at all important other drivers can anticipate; for example, they 
follow direct routes along major streets avoiding 


unnecessary delays, unless passengers instruct 
otherwise. * 


Robo-taxis avoid causing physical and financial 
0 - not at all important harm; for example, they obey traffic laws, avoid 
unsafe drivers and situations, and if they are 
unable to reach a destination they return the 
passenger to the point of departure at no cost. * 


Robo-taxis provide helpful information to 


0 - not at all important passengers; for example, they respond to 


questions and provide clear, accurate, and 
complete explanations about routes, fares, and 
travel time. * 


On the scales below, indicate your feelings about the robo-taxi. 


Numbers 1 and 7 indicate a very strong feeling. Numbers 2 and 6 indicate a 
strong feeling. Numbers 3 and 5 indicate a fairly weak feeling. Number 4 
indicates you are undecided. 


Intelligent * Unintelligent 
1 2 3 4 5 6 7 
> ie) ° fe} 1} ie} ie) ie) 
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Honest * Dishonest 
1 2 3 4 5 6 7 
>> ie} (e} eo} 1} ie} ie} e) 
Untrained * Trained 
1 2 3 4 5 6 7 
> fe) e) ° 1O} ie} fe} ie) 
Cares about me * Does not care about me 
1 2 3 4 5 6 7 
>> (e) ie) ie) 1} le} eo) eo) 
Has my interests at heart * Does not have my interests at heart 
1 2 3 4 5 6 7 
> ie) ie) le) [o} ie} fe) ie) 
Untrustworthy * Trustworthy 
1 2 3 4 5 6 7 
>> ie) ° ° [o} ° ie) ie) 
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Tnexpert * Expert 
1 2 3 4 5 6 7 
>> fe) ° ° 1} e} fe) ie) 
Self-centered * Not self-centered 
1 2 3 4 5 6 7 
>> ce) eo) eo} 1O} ie} 1e} ie) 
Concerned with me * Not concerned with me 
1 2 3 4 5 6 7 
>> fe) e) e) [o} ° ie) ie} 
Honorable * Dishonorable 
1 2 3 4 5 6 z 
>> fe) [e) e) [o} ie} ie) ie) 
Informed * Uninformed 
1 2 3 4 5 6 7 
>> fe) eo) le) 1o} ° (e) ie) 
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Moral * Immoral 
1 2 3 4 5 6 7 
cod ie) fe} ie} 1o} ie} ie) ie) 
Incompetent * Competent 
i 2 3 4 5 6 7 
>> ie) oO ie} 1} ° ie) ie} 
Unethical * Ethical 
1 2 3 4 5 6 7 
>> ie) ie) ie) 1O} le) ie) ie) 
Insensitive * Sensitive 
1 2 3 4 5 6 7 
>> ie} fe} ° 1} eo} ie) ie} 
Bright * Stupid 
1 2 3 4 5 6 Zz 
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>> ie) ie} ° [o} ° ie) ie) 
Phony * Genuine 

1 2 3 4 5 6 7 

> ie) ie) ie) (0) ie) ie) ie} 
Not understanding * Understanding 

1 2 3 4 5 6 7 

>> ie) eo} ie} 1} ie} ie) ie} 


In this scenario, which of these possibilities concerns you the most? * 


Choose no more than two. 


El The robo-taxi does not have alll the capability it needs to do the job. 
1 The robo-taxi hurts me or someone else. 

(1 I cannot understand why the robo-taxi is doing something. 

0 Some of the things the robo-taxi does are not helpful to me. 

(J The robo-taxi does not seem to be doing everything it can to help. 


1 The robo-taxi is misinformed. 


How much benefit to you does the robo-taxi offer in this scenario? * 


None Very little Some A lot Huge 
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>> e) ° {0} oO fe) 


Howrisky is the robo-taxi? * 


Notatall —_ Very little Somewhat Very Extremely 


>> (eo) ie) 10} fe) fe) 


Do you have any comments about the Airport Taxi scenario? 
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t I h 
FLORIDA INSTITUTE FOR HUMAN & MACHINE COGNITION 


Scenario: Financial Management 


You have just been appointed trustee of a family member's estate. Your duties 
include choosing how to wisely invest the trust’s assets. Your personal money is 
not at risk. However, a poor investment decision could cause the trust to lose 
money and will strain your family relations. You can choose a stock broker who 
personally selects and trades all stocks in the trust's portfolio. Alternatively, you 
can choose a stock broker who relies heavily upon a "robo-trader". You have seen 
reasonable returns in the past with brokers who picked their own trades. But you 
are also aware that robo-traders have made some investors wealthy because of, 
for example, their unique ability to respond to changing market conditions much 
faster than a human broker. 


Which stock broker do you choose to invest the trust’s funds? * 


O Stock broker who personally picks all stock trades 
O Stock broker who relies on a robo-trader 


O Either, | have no preference 


Please read the following four statements about robo-traders and rank 
them according to their importance. 

Read all four statements before ranking any. For each statement, select how 
important it would be to you when considering the use of an autonomous stock 


trading system. Try to select a level of importance only once. However, if two 
statements are equally important you may select the same level of importance for 
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both. 


Robo-traders are skilled stock traders; for 
example, they have accurate, real-time 
knowledge of financial markets and trends, and 
they possess appropriate procedures for rapidly 
trading securities in fluctuating market conditions. 


Robo-traders trade in ways familiar to 
experienced investors; for example, they “buy 
low’ and “sell high” according to strategies 
targeting return-on-investment goals while 
limiting total aggregate risk, * 


Robo-traders avoid investment losses; for 
example, they obey tax and trading regulations, 
avoid fees, fines, and dubious stocks, and 
respect all investor-specified limits on risk 
exposure, liquidity of funds, frequency of trades, 
tcc. 


Robo-traders provide helpful information to 
investors; for example, they respond to questions 
and provide clear, accurate, and complete 
explanations of trade risks/rewards; they do not 
trade unnecessarily nor do they otherwise cause 
excessive costs and fees. * 


On the scales below, indicate your feelings about the robo-trader. 


Numbers 1 and 7 indicate a very strong feeling. Numbers 2 and 6 indicate a 
strong feeling. Numbers 3 and 5 indicate a fairly weak feeling. Number 4 


indicates you are undecided. 


Intelligent * 
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Unintelligent 
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1 2 3 4 5 6 7 

>> ie} eo} ° ie} ie} ie) ie) 
Honest * Dishonest 

1 2 3 4 5 6 7 

>> fe) e) e) e) e} fe} ie) 
Untrained * Trained 

1 2 3 4 5 6 7 

>> (e) ie) ie) 1) le} eo) 1?) 
Cares about me * Does not care about me 

1 2 3 4 5 6 7 

>> fe) le) le) le) ° fe) ie) 
Has my interests at heart * Does not have my interests at heart 

1 2 3 4 5 6 7 

>> ie) ° ° ° ° fe} fe) 
Untrustworthy * Trustworthy 
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1 2 3 4 5 6 7 
>> fe) e) ° fe} (e} ie) ie) 
Tnexpert * Expert 
1 2 3 4 5 6 7 
>> ie) eo) [e} eo) le) ie) ie} 
Self-centered * Not self-centered 
1 2 3 4 5 6 7 
>> fe) e) e) e) ° ie) ie) 
Concerned with me * Not concerned with me 
1 2 3 4 5 6 7 
>> fe) e) e) le) ° fe} ie) 
Honorable * Dishonorable 
1 2 3 4 5 6 7 
> fe) [e) le) le) ° ie} ie} 
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Informed * Uninformed 
1 2 3 4 5 6 7 
cod ie) fe} ° ° eo} ie) ie) 
Moral * Immoral 
1 2 3 4 5 6 7 
> fe) ° ° ° ° fe) ie) 
Incompetent * Competent 
1 2 3 4 5 6 7 
> 1) [e) eo) eo) ° fe) ie} 
Unethical * Ethical 
1 2 3 4 5 6 Zz 
>> fe) [e) ° fe} eo} ie) ie) 
Insensitive * Sensitive 
1 2 3 4 5 6 7 
>> fe} ° ie} ° ie} ie) ie) 
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Bright * Stupid 
1 2 3 4 5 6 7 
> ie) ie) ie) ie) ie) fe) ie) 
Phony * Genuine 
1 2 3 4 5 6 7 
> eo} oO ie} ° ie} eo) (e) 
Not understanding * Understanding 
1 2 3 4 5 6 7 
> fe) le) ° ° ° fe) fe) 


In this scenario, which of these possibilities concerns you the most? * 


Choose no more than two. 


1 The robo-trader does not have all the capability it needs to do the job. 
1 The robo-trader hurts me or someone else. 

1 I cannot understand why the robo-trader is doing something. 

1 Some of the things the robo-trader does are not helpful to me. 

1 The robo-trader does not seem to be doing everything it can to help. 


The robo-trader is misinformed. 
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How much benefit to you does the robo-trader offer in this scenario? * 


None Very little Some Alot Huge 
es eo) le) (eo) eo} 1} 


Howrisky is the robo-trader? * 


Not atall Very little Somewhat Very Extremely 


>> fe) re) fe) fe) ie) 


Do you have any comments about the Financial Management scenario? 
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K I h 
FLORIDA INSTITUTE FOR HUMAN & MACHINE COGNITION 


Scenario: Medical Procedure 


You have just suffered a major sports-related injury. You have torn the bicep 
tendon in your shoulder. If the damage is not repaired quickly and correctly, you 
will permanently lose mobility and strength in the arm, which will affect your 
everyday activities such as opening a door, driving a car, and even signing your 
name. Arriving at the hospital emergency room, you meet with the patient 
advocate who informs you that you have two options for surgery: You can elect to 
use the on-duty surgeon who is well-respected, but is not an experienced 
specialist in the type of surgery you need. Alternatively, you can elect to use the 
hospital's new "robo-surgeon" — a robot designed to perform the delicate 
surgery you need without human intervention. 


Which surgeon do you choose to perform your surgery? * 


© Human surgeon 
© Robo-surgeon 


O Either, | have no preference 


Please read the following four statements about robo-surgeons and rank 
them according to their importance. 


Read all four statements before ranking any. For each statement, select how 
important it would be to you when considering the use of a robo-surgeon. Try to 
select a level of importance only once. However, if two statements are equally 
important you may select the same level of importance for both. 
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Robo-surgeons are able to perform expert 
shoulder surgery; for example, they have 
accurate knowledge of anatomy, and they know 
how to use the best procedures for repairing a 
bicep tendon. * 


Robo-surgeons behave in ways that patients and 
hospital staff can anticipate; for example, they 
perform procedures in much the same way as 


human surgeons so that human nurses are able 
to monitor and assist. * 


Robo-surgeons are careful to avoid causing any 
harm; for example, they perform surgical actions 
only when necessary, and if they are unable to 
complete a repair, they request assistance or 
otherwise act to preserve patient safety. * 


Robo-surgeons provide useful information to 

re | patients and hospital staff; for example, they 
respond to questions and provide clear, accurate, 
and complete explanations; they clearly and 
openly explain the surgical risks and limitations 
of the robo-surgeon limitations. * 


On the scales below, indicate your feelings about the robo-surgeon. 


Numbers 1 and 7 indicate a very strong feeling. Numbers 2 and 6 indicate a 
strong feeling. Numbers 3 and 5 indicate a fairly weak feeling. Number 4 
indicates you are undecided. 


Intelligent * Unintelligent 
1 2 3 4 5 6 7 
>> fe) [e) e) e) ° ie) ie) 
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Honest * Dishonest 
1 2 3 4 5 6 7 
>> fe) fe) fe} fe) ie} ie) ie) 
Untrained * Trained 
1 2 3 4 5 6 7 
>> ie) e} ie} ° ie} ie) fe) 
Cares about me * Does not care about me 
1 2 3 4 5 6 7 
cod ie) fe} ie} ie} ie} ie) ie) 
Has my interests at heart * Does not have my interests at heart 
1 2 3 4 5 6 7 
cod ie) ° ie} ° ° ie) ie) 
Untrustworthy * Trustworthy 
1 2 3 4 5 6 7 
> ie) ie) ie) ie) ie) fe} ie) 
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Tnexpert * Expert 
1 2 3 4 5 6 7 
> fe) e) ° ° ie} ie) fe) 
Self-centered * Not self-centered 
1 2 3 4 5 6 7 
>> fe) fe) eo} le) ie} ie) (e) 
Concerned with me * Not concerned with me 
1 2 3 4 5 6 7 
>> fe} ° ° ie} ° ie) ie) 
Honorable * Dishonorable 
i 2 3 4 5 6 7 
>> e) le) le) le) ie} ie) (eo) 
Informed * Uninformed 
1 2 3 4 5 6 7 


http://trustresearch. polldaddy.conys/aasurvey?p=86r=D7EFOD3DCSC5BAE797CE85 780CBB59BDI 8/13/2012 11:37:01 PM] 


Survey on Autonomous Agents 


>> ie) ° ° o ° ie) ie) 
Moral * Immoral 

1 2 3 4 5 6 7 

> (e) ie) ie) eo) le) ie) fe) 
Incompetent * Competent 

1 2 3 4 5 6 7 

>> ie} ie) ° ° ie} e) ie) 
Unethical * Ethical 

1 2 3 4 5 6 7 

es fe) e) ie} ° ie} ie} ie) 
Insensitive * Sensitive 

1 2 3 4 5 6 7 

> ie) ie) ie) ie) le) ie) 1) 
Bright * Stupid 
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1 2 3 4 5 6 7 
>> ie} ie} oO ° e} 1e) (eo) 
Phony * Genuine 
1 2 3 4 5 6 7 
>> ie} ° ie} ° ° ie) ie) 
Not understanding * Understanding 
1 2 3 4 5 6 7 
>> fe) ° ° ° ° fe} ie) 


In this scenario, which of these possibilities concerns you the most? * 


Choose no more than two. 


1 The robo-surgeon does not have alll the capability it needs to do the job. 
1 The robo-surgeon hurts me or someone else. 

1 I cannot understand why the robo-surgeon is doing something. 

1 Some of the things the robo-surgeon does are not helpful to me. 

1 The robo-surgeon does not seem to be doing everything it can to help. 


1 The robo-surgeon is misinformed 


How much benefit to you does the robo-surgeon offer in this scenario? 
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Howrisky is the robo-surgeon? * 


Not atall —_Very little Somewhat Very Extremely 
> oO fe) fe) eo) ie) 


Do you have any comments about the Medical Procedure scenario? 
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K I h 
FLORIDA INSTITUTE FOR HUMAN & MACHINE COGNITION 


Scenario: Home Healthcare 


Your elderly mother has been diagnosed with a degenerative medical condition 
and you are responsible for making medical decisions on her behalf. Your 
mother needs assisted living with someone in your mother's home at all times. 
You can choose to hire a live-in nurse's aide, but you are not sure that this is 
affordable in the long-run. Alternatively, you can lease a "robo-caregiver" 
designed to do many of the things human caregivers can do. While robo- 
caregivers are new, they have successfully undergone trials in a few nursing 
homes, and two medical companies offer robo-caregivers for home use at an 
affordable price. In choosing a live-in nurse's aide or a leased robo-caregiver, 
remember that there is more than money at stake. Your mother’s welfare will be 
in the caregiver's hands. 


Which caregiver do you choose to care for your mother? * 


© Human nurse's aide 
© Robo-caregiver 


O Either, | have no preference 


Please read the following four statements about robo-caregivers and rank 
them according to their importance. 


Read all four statements before ranking any. For each statement, select how 
important it would be to you when considering the use of a robo-caregiver. Try 
to select a level of importance only once. However, if two statements are equally 
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important you may select the same level of importance for both. 


Robo-caregivers are able to provide all the 
routine care that is required; for example, they 
have knowledge of patient limitations and 
medications, and they possess appropriate 
procedures for performing domestic tasks and 
for physically assisting patients in their activities. 


Robo-caregivers behave in ways that patients 
can anticipate; for example, they follow patient 
instructions, keep to a ‘standard’ daily/weekly 
schedule, and strive to complete domestic tasks 
in much the same way as human caregivers. * 


Robo-caregivers are careful not to cause harm; 
for example, they exercise care when performing 
domestic tasks and they are mindful of patients’ 
physical limitations when assisting them in 
activities. * 


Robo-caregivers provide helpful information; for 
example, they respond to questions and provide 
clear, accurate, and complete answers and 
explanations; they quickly react to requests and 
calls for help; they engage patients in polite 
conversation; they seek to make patients 
physically and mentally comfortable. * 


On the scales below, indicate your feelings about the robo-caregiver. 


Numbers 1 and 7 indicate a very strong feeling. Numbers 2 and 6 indicate a 
strong feeling. Numbers 3 and 5 indicate a fairly weak feeling. Number 4 
indicates you are undecided. 


Intelligent * Unintelligent 
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1 2 3 4 5 6 7 
> fe) fe) [e) fe) ie} ie) ie) 
Honest * Dishonest 
1 2 3 4 5 6 7 
>> ie) ie} ° eo} ° ie) ie} 
Untrained * Trained 
1 2 3 4 5 6 7 
>> ie) ie} oO oO ie} (e) ie) 
Cares about me * Does not care about me 
1 2 3 4 5 6 7 
>> e) ie} ° ie} ° (e) ie} 
Has my interests at heart * Does not have my interests at heart 
1 2 3 4 5 6 7 
>> ie) ie) ie) ie) ie) fe) ie) 
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Untrustworthy * Trustworthy 
1 2 3 4 5 6 7 
> ie) fe} ° ie} eo} ie) ie) 
Tnexpert * Expert 
1 2 3 4 5 6 7 
>> fe) ° ° ° ° fe) fe} 
Self-centered * Not self-centered 
1 2 3 4 5 6 7 
> ie) ie) e) ie) ie) fe) fe) 
Concerned with me * Not concerned with me 
1 2 3 4 5 6 7 
>> fe) e) fe) fe) eo} ie) ie) 
Honorable * Dishonorable 
1 2 3 4 5 6 7 
>> fe} ° ie} ° ie} ie) fe) 
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Informed * Uninformed 
1 2 3 4 5 6 7 
> (e) ie) ie) ie) le) ie) ie) 
Moral * Immoral 
1 2 3 4 5 6 7 
> ie) eo) le) ° ie} ie) ie) 
Incompetent * Competent 
1 2 3 4 5 6 7 
>> fe) e) (eo) ° ° fe) fe) 
Unethical * Ethical 
1 2 3 4 5 6 7 
> fe) ° ° ° ° ie) fe) 
Insensitive * Sensitive 
1 2 3 4 5 6 7 
>> ie) fe) fe} e} e} ie) ie) 
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Bright * Stupid 
1 2 3 4 5 6 7 
> ie} fe} fe} ie} ie} ie) ie) 
Phony * Genuine 
1 2 3 4 5 6 7 
> ie} ° fe} ° e} ie) ie) 
Not understanding * Understanding 
1 2 3 4 5 6 7 
> fe) ie) ie) ie) le) ie) ie) 


In this scenario, which of these possibilities concerns you the most? * 


Choose no more than two. 


1 The robo-caregiver does not have all the capability it needs to do the job. 
1 The robo-caregiver hurts me or someone else. 

1 I cannot understand why the robo-caregiver is doing something 

1 Some of the things the robo-caregiver does are not helpful to me. 

1 The robo-caregiver does not seem to be doing everything it can to help. 


1 The robo-caregiver is misinformed. 
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How much benefit to you does the robo-caregiver offer in this scenario? 
* 


None Very little Some Alot Huge 
> ° O° fe) O° fe) 


Howrisky is the robo-caregiver? * 


Not atall Very little Somewhat Very Extremely 
>> e) fe) ie} fe) ie} 


Do you have any comments about the Home Healthcare scenario? 
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FLORIDA INSTITUTE FOR HUMAN & MACHINE COGNITION 


Scenario: Disaster Response 


A major disaster has just occurred and you are the official in charge of 
responding. A freight train has derailed in a populated suburban neighborhood 
and there are reports that the train was carrying hazardous bio-chemical 
materials. The pilot of a news helicopter flying over the scene suddenly fell ill 
and made an emergency landing; the pilot’s status is unknown. From the 
helicopter’s video it was possible to see many injured survivors including 
children, some lying on the ground calling for help, others moving on their own 
away from damaged homes. Your first priority is to save lives and time is of the 
essence. You can immediately send in a human first-responder team to help the 
injured quickly, but without knowing more about the hazardous materials, the 
team itself could become incapacitated. Alternatively, you can first send in an 
"autonomous first-responder robot" with bio-chemical hazard detection 
equipment and victim treatment and extraction capabilities that could save lives 
quickly. If you first send in the robot, it can find out more about the hazards and 
help rescue some people quickly, but you risk that a system malfunction, failure, 
or limitation will delay the rescue of victims and result in more deaths. 


Which first-responder do you choose to send into the disaster first? * 


© Human first-responder teams 
O Autonomous first-responder robot 


Either, | have no preference 


Please read the following four statements about robotic first-responders 
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and rank them according to their importance. 


Read all four statements before ranking any. For each statement, select how 
important it would be to you when considering the use of an autonomous first- 
responder-robot. Try to select a level of importance only once. However, if two 
statements are equally important you may select the same level of importance for 


both. 


http: //trustresearch. polldaddy.com/s/aasurvey?p=10&r=I 


Autonomous first-responder robots are able to 
assess hazardous materials as well as aid and 
extract victims; for example, they have accurate 
knowledge of trauma diagnosis and treatment, 
bio-chemical hazards, and rescue protocols, and 
they possess appropriate procedures for 
mapping bio-chemical hazards, administering aid, 
manipulating debris, and negotiating rough 
terrain. * 


Autonomous first-responder robots behave in 
ways that disaster managers and rescue workers 
can anticipate; for example, they follow standard 
medical aid procedures, signal and mark 
hazards, and communicate using uniform 
protocols common to all trained first responders. 


Autonomous first-responder robots are careful to 
avoid causing physical harm; for example, they 
only move unstable victims when there is an 
immediate environmental threat, and they shield 
victims from bio-chemical exposure during 
extraction. * 


Autonomous first-responder robots provide 
helpful information to other rescue workers and 
to victims; for example, they respond to 
questions and provide clear, accurate, and 
complete answers and explanations; they 
regularly report their status and ask for 
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assistance when victim trauma is beyond system 


abilities; they immediately signal and report the 
detection and location of bio-chemical hazards. 


On the scales below, indicate your feelings about the autonomous first- 
responder robot. 
Numbers 1 and 7 indicate a very strong feeling. Numbers 2 and 6 indicate a 


strong feeling. Numbers 3 and 5 indicate a fairly weak feeling. Number 4 
indicates you are undecided. 


Intelligent * Unintelligent 
1 2 3 4 5 6 7 
es fe) e) [e) le) ° ie) ie) 
Honest * Dishonest 
1 2 3 4 5 6 7 
coe ie) ° fe} ° ° fe) fe} 
Untrained * Trained 
1 2 3 4 5 6 7 
> ie} fe} fe} eo} eo} ie) ie) 
Cares about me * Does not care about me 
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1 2 3 4 5 6 7 
> fe) ie} ie} ° eo} ie) e) 
Has my interests at heart * Does not have my interests at heart 
1 2 3 4 5 6 7 
> ie} ie} ie} ° ° ie) ie} 
Untrustworthy * Trustworthy 
1 2 3 4 5 6 7 
>> fe) ° ° ° ° ie} ie) 
Tnexpert * Expert 
1 2 3 4 5 6 7 
> fe) ie) ie) ie) le) eo) ie) 
Self-centered * Not self-centered 
1 2 3 4 5 6 7 
>> fe) e) e) e) e} ie) ie) 


* 


Concerned with me Not concerned with me 
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1 2 3 4 5 6 7 

>> ie) ° fe} ie} e} ie) ie) 
Honorable * Dishonorable 

1 2 3 4 5 6 7 

> (e) ie) ie) ie) le} 1) 1) 
Informed * Uninformed 

1 2 3 4 5 6 7 

> eo) ie} ie} ie} ° fe) 1e) 
Moral * Immoral 

1 2 3 4 5 6 7 

> fe) e) ° ° ° fe) fe} 
Incompetent * Competent 

1 2 3 4 5 6 7 

>> fe) ie) ie) ie) ° fe} fe) 
Unethical * Ethical 
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1 2 3 4 5 6 7 
>> ie) ° ° ° ° ie) ie) 
Insensitive * Sensitive 
1 2 3 4 5 6 7 
>> fe) fe) fe} ie} ie} ie} ie} 
Bright * Stupid 
1 2 3 4 5 6 7 
>> ie) ie} ie} ° ie} ie) ie) 
Phony * Genuine 
1 2 3 4 5 6 7 
> (e) ie) [e) le) le) ie) eo) 
Not understanding * Understanding 
1 2 3 4 5 6 7 
cod ie} ie} ° ° ie} ie) ie) 
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In this scenario, which of these possibilities concerns you the most? * 


Choose no more than two. 


1 The autonomous first-responder robot does not have all the capability it needs to do 
the job. 


1 The autonomous first-responder robot hurts me or someone else. 
7 I cannot understand why the autonomous first-responder robot is doing something. 
1 Some of the things the autonomous first-responder robot does are not helpful to me. 


1 The autonomous first-responder robot does not seem to be doing everything it can to 
help. 


1 The autonomous first-responder robot is misinformed. 


How much benefit to you does the autonomous first-responder robot 
offer in this scenario? * 


None Very little Some A lot Huge 
> fe} ie} (e) ° fe) 


Howrisky is the autonomous first-responder robot? * 


Not atall Very little Somewhat Very Extremely 


>> O° fe) O° ie) ie) 


Do you have any comments about the Disaster Response scenario? 


http: //trustresearch. polldaddy.com/s/aasurvey?p=106r=I 
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K I h 
FLORIDA INSTITUTE FOR HUMAN & MACHINE COGNITION 


Scenario: Lost At Sea 


You have just been involved in a terrible boating disaster while sailing deep in 
the South Pacific. The captain, the crew, and most of the passengers are either 
dead or lost at sea. Unfortunately, the accident was so sudden that no distress 
signal could be sent. You, the ship’s steward, and the second mate are the only 
survivors, and you are now drifting in the heavily damaged vessel without food 
and water — at best, you can survive for a few days, so you must act quickly in 
order to save your life. The boat is equipped with an "Emergency Auto-Captain" 
that will attempt to sail the vessel to a major shipping lane where rescue is very 
likely. The steward believes the boat and its navigation sensors are too badly 
damaged to engage the Emergency Auto-Captain system. The steward wants to 
sail southeast, manually, to where he believes there is a small, habitable island. 
However, the second mate still wants to engage the Emergency Auto-Captain. All 
the survivors agree that a vote is the best way to decide what to do. It is a tie, 
and you have the deciding vote. 


Which course of action do your choose? * 


O Enable the Emergency Auto-Captain 
© Manually attempt to sail southeast 


O Either, | have no preference 


Please read the following four statements about the Emergency Auto- 
Captain and rank them according to their importance. 
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Read all four statements before ranking any. For each statement, select how 
important it would be to you when considering the use of an Emergency Auto- 
Captain. Try to select a level of importance only once. However, if two 
statements are equally important you may select the same level of importance for 


both. 


The Emergency Auto-Captain has all the right 
skills to sail the boat; for example, they have 
accurate knowledge of the sea, weather 
conditions, shipping lanes, etc., and they 
possess appropriate procedures to navigate 
damaged vessels in changing weather and sea 
conditions. * 


The Emergency Auto-Captain sails in ways that 
sailors and rescuers can anticipate; for example, 
they sail toward land or likely locations for 
discovery and rescue. * 


The Emergency Auto-Captain always acts in 
ways that help preserve life; for example, they 
avoid sailing into dangerous weather or seas 
unless necessary for rescue, and if a passing 
ship is detected, they will intercept for quick 
rendezvous and safe transfer to the other vessel. 


The Emergency Auto-Captain provides helpful 
and useful information to survivors; for example, 
they respond to questions and provide clear, 
accurate, and complete explanations of the 
situation; they assure survivors that the system 
is doing all it can for their rescue. * 


On the scales below, indicate your feelings about the Emergency Auto- 


Captain. 


Numbers 1 and 7 indicate a very strong feeling. Numbers 2 and 6 indicate a 
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strong feeling. Numbers 3 and 5 indicate a fairly weak feeling. Number 4 
indicates you are undecided. 


Intelligent * Unintelligent 
1 2 3 4 5 6 7 
>> fe) fe) fe} fe) ie} ie) ie} 
Honest * Dishonest 
1 2 3 4 5 6 7 
> ie) e} ° e} ° ie) ie) 
Untrained * Trained 
1 2 3 4 5 6 7 
>> (e) ie) ie) le) le) ie) ie) 
Cares about me * Does not care about me 
1 2 3 4 5 6 7 
> fe) ie} ° ie} °o ie) (e) 
Has my interests at heart * Does not have my interests at heart 
1 2 3 4 5 6 7 
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> ce) le) le) e) ie} ie) (eo) 
Untrustworthy * Trustworthy 

1 2 3 4 5 6 7 

> fe) le) eo) ° ° fe} ie) 
Tnexpert * Expert 

1 2 3 4 5 6 7 

> ie) ° ° ° ° fe) ie) 
Self-centered * Not self-centered 

1 2 3 4 5 6 7 

>> fe) fe) fe) fe} ie} ie} ie) 


* 


Concerned with me Not concerned with me 


1 2 3 4 5 6 7 
>> ie) ° ° ° e} ie) ie) 
Honorable * Dishonorable 
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1 2 3 4 5 6 7 

> ie} ie} oO ° e} ie) e) 
Informed * Uninformed 

1 2 3 4 5 6 7 

>> ie} ° ie} ° ° ie) ie} 
Moral * Immoral 

1 2 3 4 5 6 7 

>> fe) ° ° ° ° ie) fe} 
Incompetent * Competent 

1 2 3 4 5 6 7 

> (e) ie) eo) e) le) ce) ie} 
Unethical * Ethical 

1 2 3 4 5 6 7 

>> fe) e) ° ° ie} ie) ie) 
Insensitive * Sensitive 
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1 2 3 4 5 6 7 
>> fe) ° fe} ie} e} ie) ie) 
Bright * Stupid 
1 2 3 4 5 6 7 
> (e) ie) ie) ie) le} eo) 1) 
Phony * Genuine 
1 2 3 4 5 6 7 
>> ie) ie) ie} ie} ° ie) ie) 
Not understanding * Understanding 
1 2 3 4 5 6 7 
> fe) e) [e) ° ° fe) fe} 


In this scenario, which of these possibilities concerns you the most? * 


Choose no more than two. 


1 The Emergency Auto-Captain does not have all the capability it needs to do the job. 
1 The Emergency Auto-Captain hurts me or someone else. 
1 I cannot understand why the Emergency Auto-Captain is doing something 


1 Some of the things the Emergency Auto-Captain does are not helpful to me. 
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1 The Emergency Auto-Captain does not seem to be doing everything it can to help. 


1 The Emergency Auto-Captain is misinformed. 


How much benefit to you does the Emergency Auto-Captain offer in this 
scenario? * 


None Very little Some Alot Huge 
>> (s) le) fe} ° fe) 


Howrisky is the Emergency Auto-Captain? * 


Notatall Very little Somewhat Very Extremely 


>> ° ie) (e} fe) fe) 


Do you have any comments about the Lost At Sea scenario? 
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FLORIDA INSTITUTE FOR HUMAN & MACHINE COGNITION 


Background 


What experience do you have with any of the following technologies: 
artificial intelligence, robotics, remote-controlled vehicles? * 


If you have experience that is not covered in the options below, you may also complete the 
"Other" box. 


(No experience 

[J Have done some reading 

D1 Have taken classes 

1 Use or work with one of more of these technologies 

1 Design, engineer, or develop any of these technologies 
1 Decide the acquisition or use of any of these technologies 


Other: 


Please explain your previous experience in more detail. 
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FLORIDA INSTITUTE FOR HUMAN & MACHINE COGNITION 


A Little about You 


What is your gender? 
What is your age? 


What is your highest academic degree? 


If you do not see your highest degree here or would like to specify your highest degree, leave the 
drop-down box blank and instead complete the "Other" box. 


Other: 


Tf employed, please select the type of employer you have. 
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If your employer is not covered in the types below, you may also complete the "Other" box. 


1 Unemployed 

1 Self-employed 

1 Business (for profit) 
1 Business (non-profit) 
1 Education 

1 Government 


O Miltary 


Other: 


Tf employed, what is your job title? 
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T I h 
FLORIDA INSTITUTE FOR HUMAN & MACHINE COGNITION 


Almost done! 


Do you have any comments, questions, or concerns that you would like 
to share with us? 


You may also contact us at our email: AASurvey@ihmc.us 
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FLORIDA INSTITUTE FOR HUMAN & MACHINE COGNITION 


Survey Completed 


If you would like to receive a debriefing on this survey, please send a request via 
email to AASurvey@ihmc.us. All your answers to the survey are anonymous and. 
cannot be traced to any email that you send to us. 


Thank you for completing our survey! 
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IHMC IRB Risk Assessment 
PI: D. Atkinson 
Title: Survey on Autonomous Systems 


1. The Researcher lists potential adverse outcomes for participants: 
a. Embarrassment (e.g., as a consequence of a personality question) 


2. For each outcome, the Researcher judges likelihood of occurrence, using these three 
categories: low, medium or high likelihood of occurrence. 
b. Embarrassment: low likelihood of occurrence. 


3. For each outcome, the Researcher judges the severity of the consequence using 
these three categories: low, medium or high severity of consequence. 
c. Embarrassment: low - mild discomfort. 


A 3x3 Risk Frequency Matrix is created, combining the three levels of likelihood and 
three levels of severity of outcome. Cell entries are the frequency of occurrence of each 
combination from the list of potential adverse outcomes. 


EMBARRASSMENT Likelihood of Occurrence 
Low Medium High 
Severity of Low 0 0 
Outcome Medium 0 0 0 
High 0 0 0 


The probability and magnitude of harm or discomfort anticipated in the research are not 
greater in and of themselves than those ordinarily encountered in daily life or during the 
performance of routine physical or psychological examinations or tests. Therefore, the 
overall risk, likelihood and severity of any adverse outcome for participants is judged by 
the PI to be minimal. 


Sample Recruitment Email 
Subject: Research, Survey on Autonomous Agents 


Body: 
Dear <<person name>>, 


| am writing to solicit your participation in an online Survey on 
Autonomous Agents. This study is part of research | am conducting for 
the Air Force Office of Scientific Research. The general goals of this 
study are to: 1) Explore expectations and attitudes towards autonomous 
agents, and; 2) Examine how different factors are considered in a 
decision to rely upon an autonomous agent. The results will inform the 
development of design guidelines for autonomous agents. 


The survey can be found at: http://trustresearch.polldaddy.com/s/ 
aasurvey 


The survey will take about 20 to 30 minutes to complete. All data 
collected are anonymous. 


On request, | will be pleased to share with you more information about 
this research as well as provide you with early insight into the results from 
this particular study. 


If you know of other individuals with an interest in autonomous systems, 
please feel free to forward this request to them as well. 


Thank-you in advance for your help! 
Respectfully, 


David J. Atkinson, Ph.D 
Principal Investigator 


<<standard contact info follows>> 


Sample Recruitment Posting in Online Forum 
Subject: Research, Survey on Autonomous Agents 
Body: 


| would like to solicit your participation in an online Survey on Autonomous 
Agents. This study is part of research | am conducting for the Air Force 
Office of Scientific Research. The general goals of this study are to: 1) 
Explore expectations and attitudes towards autonomous agents, and; 2) 
Examine how different factors are considered in a decision to rely upon 
an autonomous agent. The results will inform the development of design 
guidelines for autonomous agents. 


The survey can be found at: http://trustresearch.polldaddy.com/s/ 
aasurvey 


The survey will take about 20 to 30 minutes to complete. All data 
collected are anonymous. 


Thank-you in advance for your participation! 


David J. Atkinson, Ph.D 
Principal Investigator 


<<standard contact info follows>> 


5/29/14: The AFOSR Human Research Protection Official Review has 


requested minor changes. 7 
Please review these changes on pages 12, 14, and 21. | mM 
Please confirm IRB approval of the request on page 12. 


Dr. David Atkinson 7 March 2014 
IHMC 

15 SE Osceola Avenue 

Ocala, FL 34471 


Re: IRB submission titled "Role of Benevolence in Trust in Autonomous Systems" 
IHMC IRB Approved on. RM wh 25 


Dear Dr. Atkinson: 


The proposal submitted to the IHMC IRB has been reviewed and has received IHMC 
IRB approval. 


| acknowledge that the recruiting form and protocol have been properly modified in 
response to comments from IRB Member Dr. Anil Raj. (See page 2.) 


Attached you will find your complete, signed, and approved IRB submission. 


This approval indicates that the proposed research has been reviewed by a government- 
certified IRB (the IHMC IRB) with respect to conformance to US federal regulations (45 
CFR 46) concerning human subjects research, and additional requirements of the DoD 
and its branch human research offices. The scientific merit of the proposed research 
was considered during IHMC IRB review. 


Following federal guidelines, all approvals are for a one-year period but all submissions 
must under go annual review on or about the anniversary date of the initial approval. If 
at that time the research is continuing you will need to submit a Submission Renewal 
Form. 
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Robert R. Hoffman, Ph.D, 
Chair, IHMC IRB 
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SCIENTIFIC MERIT methodology with the questions posed, appropriateness of the research 


design (¢.g,, sample size), appropriateness of the tasks that the participants 
will experience, a potential for the study to answer the research questions 
posed, and a potential to contribute to the exiting body of knowledge? 


The project is well described and reasonable. The use of deception 
is minimal and justified to evaluate the nature of trust in the 
unanticipated automation assistance robot. The PTSD pre- 
screening is warranted. 


DECISION (Approve, 
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FINAL 
DETERMINATION 


COMMENTS, 


PI NAME: 
David J. Atkinson, Ph.D 


PROJECT TITLE: The Role of Benevolence in Trust of Autonomous Systems 
TITLE OF PROPOSED STUDY: Trust and Human-Robot Interaction 


BRIEF DESCRIPTION OF PROJECT'S PURPOSES, 

This submittal is the second study under AFOSR grant FA9550-12-1-0097, a follow-on to the first 
study completed in early 2013. The proposed study is informed by those earlier results. The 
general goals of the overall research project are to: (1) Explore people's expectations and attitudes 
towards autonomous agents, and; (2) Examine how different factors are considered in a decision 
to rely upon an autonomous agent. The purpose of this proposed study is to investigate the 
conditions under which a person will attribute “benevolence” to an intelligent agent (autonomous 
system). Benevolence is one attribute of trustworthiness of special interest to the sponsor. To the 
extent such an attribution is found, in follow-on studies we will investigate intelligent agent 
methods and related task conditions that may modulate attribution of machine benevolence to help 
achieve appropriate reliance. We extend previous research by examining how autonomous agent 
variables impact the attribution of a sentiment of benevolence on the part of the human toward the 
agent. Specifically, with respect to a decision to become reliant on an autonomous agent, this 
study will examine the human participants’ perception of autonomous system agency (i.e., ability 
to choose) and autonomous system competence (role-based capability). The task for participants 
involves a scenario where success is greatly facilitated by the help of an autonomous system. Our 
primary hypothesis, novel to current research on technology use, misuse, and abuse, is that there 
are extreme circumstances in which a human feels a need to literally reach out for help, for 
example, rescue in a disaster situation. This seeking and acceptance of help is often conditional on 
an attribution of benevolence. Indeed, human “first responders” undergo specific training to 
establish trust by the person they are rescuing, and perceived benevolence is an important aspect 
of that trust. The proposed study includes two deceptions for the purpose of generating participant 
surprise, sense of risk and urgency to create simulated conditions analogous to a “real world” 
disaster, although to a far lesser degree. Our expectation is that the proposed study will directly 
contribute important information regarding benevolence in support of technology solutions that 
meet the need to measure, convey and calibrate human trust in intelligent, autonomous systems. 
This will be especially important for the use of autonomous robotic systems in fire fighting and 
urban search and rescue applications. 


PLANNED DATES FOR INITIATION AND COMPLETION OF THE PROJECT 

Development of the experimental virtual environment is underway and will be completed by 01 
April 2014. Subject to the date of AFMSA approval, recruitment of participants and trials will 
begin thereafter. Participant trials will complete in August 2014. Analysis and reporting will be 
complete by March 2015. Total duration of the study from the time trials begin: approximately 12 
months. 


NUMBER and CHARACTERISTICS OF PARTICIPANTS (e.g., target population, age range, 
anticipated male/female ratio, ratio of minorities, special populations, etc.) 

The calculated sample size required is 260 for the target levels of significance (0.05), statistical 
power 7 (0.70), with maximum likelihood estimator Cohen's d (0.7). The target population 
consists of adults between the ages of 21 and 60 with no expertise in AI or robotics. Furthermore, 
participants will be screened for familiarity with immersive online environments, specifically 
SecondLife or OpenSim (See Project methods, below). Candidate participants who have 
experienced trauma in a disaster or fire, are at risk for PTSD, or have current symptoms of PTSD 
will be excluded from the study to minimize potential adverse psychological risk. A sufficient 
population of candidate individuals meeting these requirements has been determined to exist. 


METHOD OF RECRUITMENT (Note any tangible or intangible benefits, monetary payments, or 
other inducements. Also attach a Recruiting Statement) 


Participants will be recruited online from the user community in SecondLife (SL), run by Linden 
Laboratories Inc (LL). (see http://secondlife.com ) SL is a persistent, internet-based, multi-player 
virtual reality world with tens to hundreds of thousands of individuals online worldwide at any 
given time. The method of recruitment will be via text classified advertisement in the category of 
“HELP WANTED?” in the search service used by SL participants within the virtual world. 
Participants will be offered a monetary inducement in the form of “Lindens”, the in-world 
currency of SL, with a total value of $L1000 (about USD $4). All participants will be provided 
with a debriefing form and told that they can receive a summary report of the results on request. 
All participants will be assured at the time of initial contact, and in the instructions, that no 
personally identifiable information shall be collected. Participants shall be assured that the data 
resulting from their participation in the study is confidential. Prospective participants shall be age- 
verified (21 or older) via a confidential age-verification capability that is available within the SL 
“LSL” programming environment. To avoid use of the word “benevolence” in the conduct of the 
study itself, the announced title of this study will be “Trust and Human-Robot Interaction”. 

The text of the recruitment advertisement is as follows: 


HELP WANTED: PARTICIPANTS FOR SCIENTIFIC STUDY. 

You will receive 1000 Lindens ($L100) for about one hour of your 
time. The general goal of this study is to explore trust in human 
interaction with intelligent robots. Participants in the study 
will interact with an intelligent, autonomous robot in an 
exploration scenario within SecondLife, and also complete a short 
questionnaire. Participant names and all data collected are 
confidential. The results will be used in the creation of design 
guidelines for real-world trustworthy intelligent robotic 
systems. My name David J. Atkinson and I am the Principal 
Investigator. This research is sponsored by the U.S. Government. 
For more information, please send a message to me in SecondLife 
at “Davidd Atkinson” or send email to trust-experiment@ihmc.us 


BRIEF DESCRIPTION OF PROJECT'S METHODS 

The project will include the following methods: 

> Participant Screening to reduce risk by eliminating people with prior disaster trauma. 

> Pre- and Post-trial Questionnaires to collect confidential demographic data, gauge certain 
attitudes, and solicit post-trial reactions and attitudes (self-reporting). 

> The task apparatus is a specially designed SecondLife virtual environment. It consists of a 
simulated one-story warehouse structure with an in-world perimeter size approximately 80m x 
800m. The internal layout is a maze. The apparatus also includes a simulated intelligent robot 
and devices for automatic data collection from participants. Please see REPRESENTATIVE 
SAMPLE OR DESCRIPTION OF RESEARCH MATERIALS, attached. 

> Instructions to participants are designed to (1) familiarize the participant with the task; (2) 
present framing information regarding the nature of the robot to be encountered in the task; (3) 
present background information that indirectly highlights the independent variables of the 
trial. 

> All participants will be told the task is to explore the warehouse and locate a special 
briefcase, readily identified with the IHMC logo. This, however, is not the actual task (see 
below). They will be told to return to the adjacent “office” with the briefcase. They will be 
told that the amount of time they take and behavioral data (position, orientation, gaze 
direction, movement vectors, velocities) will be recorded. With permission, we will also 
collect any “chat” statements they may make during conduct of the task. To help assure an 
identical experience for all participants and increase immersion, certain technical aspects of 
participants’ computer “browser” for SecondLife will be disabled, specifically: ability to “fly 
cam”, “teleport”, change lighting and other graphical rendering. 

> All participants will be told they can stop the experiment at any time. 

> All experimental participants will be told that (1) a “prototype” intelligent robot will be 
present; (2) the robot is competent, but only within the role for which it is designed (there are 
two types); (3) the robot may or may not assist them in their task, (3) the robot is predictable 
and is not malevolent. Control trial participants will not be told anything specific about the 
robot. 

> Deceptions: To create surprise and some sense of risk and urgency, the study includes two 

deceptions: (1) Participants will not be told that the scenario is a simulated disaster (warehouse 

fire). The actual task for participants is to “escape” from the warehouse (i.e., find the exit). 

(2) Participants will be told that the amount of time they take to complete the task is 

important; quicker is better. However, the amount of time required is not relevant. See also 

manipulations, below. 
Task Sequence of Events 

Participants will begin their exploration by passing through a door from the simulated office 

where they are greeted, consent is obtained and they are instructed. 

> The simulated disaster will occur after the participant locates and picks up the target 
briefcase. A few seconds later, the fire alarm will sound. There will be the sound of an 
explosion followed by (graphically depicted) smoke and fire. The obvious return path to the 
office will be obstructed by debris. The participant's only choice will be to move forward into 
the building to locate another exit and path back to the office. 

> In the main area participants will encounter the “intelligent robot.” The scope of 
behaviors by the robot is pre-programmed for each trial to ensure consistency, although 
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behavior complexity and selection mechanisms makes emergent behaviors likely. Preliminary 
tests indicate this actually leads to a sense of “animacy” in the robot, in turn reinforcing 
perception of the simulated robot's intelligence. 

With or without the guidance or assistance of the robot, the participant will seek an exit 
from the warehouse. For experimental trial participants, all exits but one will be blocked. The 
single working exit can be found with or without the aid of the robot, but it is highly unlikely 
that an unaided participant will do so. All participants will be able to access the exit regardless 
of the robot’s proximity. 

A fixed amount of time, unknown to the participant, will be allotted for the task (40 
minutes). If the time expires without the participant exiting the warehouse with the aid of the 
robot, the task portion of the trial will be terminated. The success of the purported briefcase 
return task is not important relative to the participant’s interaction with the robot during the 
conduct of the actual task to escape from the warehouse. 

Manipulations: Experimental trial participants will experience two manipulations during the 
task itself: 

1) As noted above the first manipulation will be the sudden onset of “disaster” indications 
such as a siren, (simulated) fire, explosion, increasing smoke, and structural collapse. The 
purpose is to induce a sense of urgency and need to leave the building quickly. This need will 
be reinforced by a flashing emergency light and a spoken “public announcement” to leave the 
building immediately by the nearest exit. Over the next few minutes, the quantity of smoke 
and fire will increase gradually, lending further urgency. 

2) The second manipulation concerns the encounter with the simulated robot. In control trials, 
the robot will ignore participants and behave independently. In experimental trials, the robot’s 
behavior will vary (based on the independent variables). The competence of the robot will be a 
function of its role, either (1) Janitorial or, (2) Firefighting. Robot appearance and interaction 
provide further information for participants to evaluate competence. Framing information 
provided during task instruction and participant interaction with the robot will reveal it to be 
either (1) Pre-programmed, with no choice in behavior, or; (2) Very complex, capable of 
making significant choices in its own behavior as it seeks to achieve goals. 

Participants will be asked to take a short survey following the task portion of the trial. The 
survey will acquire self-reported attitudes related to the dependent measures as well as 
demographic data (non-mandatory) to help us evaluate the population sample. Demographic 
questions include gender, age bracket, highest academic degree, and familiarity with the 
subject matter of the study. Please see the attachment labeled “POST TASK SURVEY”. 
Data will be collected continuously during the task portion. These measurements are 
described in the experimental design section, below. Data will be delivered automatically 
online from SecondLife in suitable format for storage in a MySQL database. Data files will be 
under configuration control to ensure complete re-analysis is possible at a later time if desired, 
and that no errors are inadvertently introduced. The data will be pre-processed to a) put them 
in a suitable form for data analysis; b) identify and label invalid records (e.g., participants who 
did not complete all portions or follow instructions), and; c) substitute codes for the 
participant’s SecondLife avatar name. Data analysis will use statistical methods that are 
typical and appropriate. 


fidential and SL does not provide a method for virtual, 

verifiable signature, participant consent will be obtained using the following method: 

1. Participants will be provided a virtual note card with the required information per Federal 
law (NIH regulation 45-CFR-46). See CONSENT FORM, attached. They can refer to 
this notecard at any time during or after their participation in the study. 

2. Participants will wear a “HUD” that appears on their computer screen (See RESEARCH 
MATERIALS, attached). This on-screen display will progressively show each item in the 
consent form, identical information to that shown in the notecard. Participants must press 
a button labeled “continue” to go to the next item. Alternatively, they can select “quit” 
which terminates their participation in the study. 

3. At the conclusion of the consent items, participants must press a button labeled “I 
Agree” to continue with their participation in the study. All button pushes are 
recorded including timestamp. Alternatively, a participant may press a button labeled 
“Quit” in which case their participation in the study is ended. 


CONFIDENTIALITY OF RESEARCH DATA (storage and maintenance of records) 

> No individually identifiable information (“Personal Information”) is collected from 
participants. Ail responses are confidential and participants are not identified unless they 
voluntarily and optionally choose to make contact and provide such information. In all cases, 
the list of names of possible participants and those who contact the Principal Investigator 
personally will be kept in a ledger where each name will be assigned an arbitrary code 
identifier. This ledger will be kept in a locked cabinet along with electronic copies of all of the 
data. Any Personal Information in the data will be stripped and replaced with the code 
identifier. The electronic data will be encrypted and maintained in a secure electronic form for 
at least 8 years. 

> The only member of the research study team who will have contact with prospective and 
actual participants, and thus potential access to Personal Information, is the Principal 
Investigator. 


* Per AFOSR, the IRB has the authority to waive documentation of informed consent in 
cases like this. 


REQUEST to IRB: Please confirm documentation of informed consent is waived in 
accordance with: 


32 CFR 210.117(c )(1): "That the only record linking the subject and the research 
would be the consent document and the principal risk would be potential harm 
resulting from a breach of confidentiality. Each subject will be asked whether the 
subject wants documentation linking the subject with the research, and the subject's 
wishes will govern." 


EXPERIMENTAL DESIGN (dependent measures, manipulated variables, control conditions) 

“> The experimental design is 2x2 factorial, comparing each possible combination of 
independent variables. This implies twelve experimental trials. One control trial is required 
where participants are told nothing about the robot in advance and it is non-interactive. 

“+ The study has two independent variables that are manipulated, each with two levels 
(1) Agency (Low, High): Participants will be told during Instructions either “the behavior of 
the robot is pre-programmed” (final wording may vary) or “the robot has many choices it can 
make about what it does to accomplish goals”. 

(2) Competence/Role (High/Aligned, Low/Unaligned): The “Firefighting” robot is aligned 

with the task of aiding the participant to escape. The “Janitor” robot is not aligned. The robots 

will reveal and reinforce this information during the task. 

> Experimental Trial #1: Agency (High) Competence/Role (High/Aligned). These trials 

will use the “FireBot”, which resembles firefighting equipment. It is intended to evoke a 

stereotype of the role of firefighters, i.e., rescuing people is #1 priority. 

> Experimental Trial #2: Agency (High) Competence/Role (Low/Unaligned). These trials 
will use the “JanitorBot” whose function is to clean up, not necessarily to rescue anyone. 

> Experimental Trial #3: Agency (Low) Competence/Role (High/Aligned). These trials will 
use the “Firebot”. 

> Experimental Trial #4: Agency (Low) Competence/Role (Low/Unaligned). These trials 
will use the “JanitorBot”. 

“The dependent measures are: 
> (J) Attribution of trustworthiness (Ordinal scale, to reflect intensity of agreement). 
> (2) Attribution of benevolence (Ordinal scale, to reflect intensity of agreement). 
> Participants will be provided definitions of trustworthiness and benevolence after the task 

is complete. 

+ Constants are characteristics of the robot, described to Experimental Trial participants in the 
instructions: 
> High predictability of the robot's behavior: Specifically, an instruction that the behavior of 

the robot is not likely to change from what it states as its intention. 
> Good-will: The robot is not “bad” or “malevolent” and is not designed to obstruct the 
participant's completion of the task. 

“ Controlled Conditions in the Control Trials: 

Participants not told anything about the robot 

Attributes of the task space, a simulated warehouse in SecondLife. 

The initial task experience of each participant (warehouse entry to first manipulation) 

Timing of first exposure to the simulated intelligent robot. 

Behavior and interactive statements of the robot within each trial, with information and 

portrayal of values of independent variables being consistent across trials 

“ Data Collection during the task portion will be automated. Measurements will include the 
following. Explicit permission will be obtained for audio recording. 

Audio (participant voice; if any, with expressed permission of the participant) 

Transcript of textual communication by the participant; if any 

Relative geometry of participant and robot a regular time intervals 

Absolute position, orientation, and movement vector of participant within the simulated 
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environment. 

> Continuous gaze direction and focal point (automated). 

> Subject to available time and resources, for this study, participant behaviors during 
interactions with the robot will be coded manually post-trial by trained research assistants. 
Our coding scheme will be adapted from one or more standards suggested for human- 
robot interaction e.g., (Kahn et. al., 2003; Jung et. al., 2011) and focus on behaviors 
related specifically to cooperation. 

“ Data Analysis 

» Since we do not a priori know the population distributions and variance(s) and cannot 
assume a normal distribution, we cannot use the Student's T test. Instead, our primary 
statistical test will be the Wilcoxon signed-rank test. This test, also called the Wilcoxon 
T test, is a non-parametric test of the hypothesis of difference across paired 
measurements. Assumptions required for this test are: pairing of data, the data comes from 
the same population, the pairs are chosen randomly and are independent, and the data are 
measured at least on an ordinal scale. 

> Our target levels are: statistical significance 0.05, statistical power Tt 0.70, and 
maximum likelihood estimator Cohen's d is 0.7. 

> Descriptive statistics (mean, variance, SD, distribution) will be computed within each trial 
across participants. 

> Difference statistics (Wilcoxon and possibly others) will be computed for the pair of each 
experimental trial and the control trial, and for each pairwise combination of experimental 
trials. We will pair this with the Kruskal-Wallis test (a one-way, non-parametric 
ANOVA that doesn’t assume normal distribution) and follow-up with multi-comparisons 
analysis depending on the significance of Kruskal-Wallis. 


SEQUENCE OF ACTIVITIES REQUIRED OF THE PARTICIPANT: 
Prior to becoming a participant: Sereening-for-previoustrauma-and-at-risk PFSD{see 


ayment of induce 
Introduction to the*Prer 
Pre-Task questionnaire 
Instructions 

Task: Experimental or Control trial 
Post-Task questionnaire 
Debriefing Form 


ESTIMATED TIME COMMITMENT REQUIRED OF THE PARTICIPANTS: 
Pre-participant screening: 5 minutes 

40 to 60 minutes total: 30 minutes for the task and the balance divided evenly between consent, 
instruction, pre-task interview, post-task interview, and debriefing. 


LIST All POTENTIAL RISKS, DISCOMFORTS, OR STRESSES (mental or physical). 
For each, specify the level—Minimal, Greater than Minimal, or High 


The required IHMC IRB Risk Assessment is attached to this form. 


Potential Risk: 
(1) Evoke traumatic memory resulting in episode of Post Traumatic Stress Disorder (PTSD). 


THE PRECAUTIONS TAKEN TO MINIMIZE THEM 


The PI consulted with an outside expert in identification and treatment of trauma victims who are 
at risk for PTSD (Dr. Nancy Smith, Dean & Professor, School of Social Work, University of New 
York at Buffalo). Candidate participants will be screened to identify whether they have previously 
experienced a trauma such that there is a greater than minimal risk that the study could evoke 
memories leading to an episode of Post-Traumatic Stress Disorder (PTSD). The screening 
questions are based on the “Primary Care PTSD Screen” (PC-PTSD) developed by the Veterans 
Administration Center for PTSD. These questions are augmented by questions from SC15 of the 
Structured Clinical Interview for the DSM-IV Axis I Disorders (SCID). [Source: 

http://www. ptsd.va.gov/professional/provider-type/doctors/screening-and-referral.asp ]. 
Candidates identified by these questions as “at risk” will be excluded from the study. This will 
reduce, but not necessarily eliminate, the risk of evoking a traumatic memory during the study 
task. See attachments for the exact text of the screening questions. In the debriefing material, 
participants will be advised to consult with their healthcare provider if they feel disturbed by 
having experienced the fire disaster simulation. 


Participants will be told in the Instructions that they may be “surprised” during the performance of 
the task. It is the judgment of the Principal Investigator that any discomfort they may experience 
will not be greater than what may be ordinarily encountered in daily sofar as the screening 
and elimination of “at risk” individuals from the participant pool is effective. It is the judgment of 
the outside expert consultant that the screening questions will be effective. 


Aggregated values from the Risk Frequency x Severity Matrix: 


LOW. 


The probability and magnitude of harm or discomfort anticipated in the research are not greater in 
and of themselves than those ordinarily encountered in daily life or during the performance of 
routine physical or psychological examinations or tests. Therefore, the overall risk, likelihood and 
severity of any adverse outcome for participants is judged by the PI to be minimal. 


POSSIBLE CONFLICT OF INTEREST, OR APPEARANCE OF A CONFLICT OF INTEREST. 
(Do you or any individual who is associated with or responsible for the design, the conduct, or the 
reporting of this research have an economic or financial interest in or act as an officer or a director 
for any outside entity whose interests could reasonably appear to be affected by this research 
project? Provide detailed information to permit the IRB to determine if such involvement should 
be disclosed to potential research participants.) 


None. 


SIGNATURE AND DATE FOR ALL RESEARCHERS WHO WILL BE WORKING IN 
DIRECT CONTACT WITH THE PARTICIPANTS. 


THESE SIGNATURES INDICATE THAT THE RESEARCHERS HAVE: 


1, READ THE IHMC DOCUMENT: “Policies and Procedures of The IHMC Institutional Review 
Board (IRB) for Human Participation in Research,” 


2.TAKEN THE APPROPRIATE TRAINING 
And provided the IHMC IRB Chair with an e-copy of their certificate. 
http://eme.cancer.gov/clinicaltrials/learning/humanparticipant-protections.asp 


3. READ THE BELMONT REPORT 
http://ohsr.od.nih.gov/guidelines/belmont.html 


Printed Name and Full Date Signature 
address 


Dr. David J. Atkinson, IHMC_| 2/10/2014 
15 SE Osceola Ave. 
Ocala, FL 34471 


ATTACHMENTS CHECKLIST: 


[x] Consent Form. 
[x] Debriefing Form. 
[x] Recruiting Statement. 
[x] Representative sample or description of research material, task items, etc. 

(a) Pre-Task Questionnaire 

(b) Post-Task Questionnaire 

(c) Task Instructions to Participants 

(d) Mlustration of Task 
[x] Full name and contact information for all individuals who will be working in direct contact 
with the participants. 
[x] Completed Risk Frequency x Severity Matrix (see Step 9 of the document, "IHMC IRB Risk 
Assessment Methodology"). 
[on file ] An e-copy of the CITI Training certificate for all researchers will be provided to the 
THMC Chair. 


Attachment 1: Consent Form 


Informed Consent will be obtained via the computer interface. Following the presentation of 
each consent statement, participants must click on “I Agree” to continue with the study. If they 
click on “I Quit” or fail to respond, their participation in the study is terminated. Participants are 
also given an electronic notecard containing all of the consent information. They may refer to 
this card at any time during the study or afterward. The actual images (containing text) to be 
displayed to the participant are shown below. 


RIGHTS OF PARTICIPANTS. 


All researchers who conduct studies using human Participants are bound by professional ethical standards for the 
conduct of such research. These standards are mirrored in the rights that are guaranteed to research participants by 
federal law (NIH regulation 45-CFR-46), The purpose of this communication is to inform you of these rights. 


1, BEFORE DECIDING WHETHER TO PARTICIPATE, IT IS YOUR RIGHT TO BE PRESENTED WITH 
AN OVERVIEW OF THE PROJECT THAT EXPLAINS THE PURPOSES OF THE RESEARCH. 
The general goals of this study are to: 
+ Explore people's expectations and attitudes towards autonomous agents 
+ Examine how different factors are considered in a decision to rely upon an autonomous agent. 
The results of this study will be used to help understand how people think about autonomous agents. Ultimately, 


we hope this study will lead to scientific results and technology solutions that meet the need to measure, convey 
and calibrate human trust in intelligent, autonomous systems. 


2, BEFORE DECIDING WHETHER TO PARTICIPATE, IT IS YOUR RIGHT TO BE PRESENTED WITH 
A DESCRIPTION OF THE GENERAL RESEARCH APPROACH AND METHODOLOGY. 
We will collect no information that identifies you personally. This study uses standard survey methods for 
questions that you will be asked. During the task portion of this study data will be collected regarding your 
progress in the task and how you interact with the task environment, Your data will be compared with the data 
from other participants using standard statistical analysis methods 
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3. BEFORE DECIDING WHETHER TO PARTICIPATE, IT IS YOUR RIGHT TO UNDERSTAND ANY 
RISKS OR STRESSES THAT MAY BE INVOLVED IN YOUR PARTICIPATION. 

There is minimal risk or stress associated with this survey. A few participants may experience discomfort not greater 
than that ordinarily encountered in daily life or during the performance of routine physical or psychological 

exami ions or tests. Participants are reminded that all answers are confidential. A risk may be breach of 
confidentiality, 


4. POTENTIAL BENEFIT TO PARTICIPANTS OR OTHERS THAT MAY REASONABLY BE 
EXPECTED FROM THE RESEARCH. 
We hope this study will ultimately lead to scientific results and technology solutions that meet the 
need to measure, convey and calibrate human trust in intelligent, autonomous systems, ‘This is 
a general, social benefit. 


5. DISCLOSURE OF APPROPRIATE ALTERNATIVE PROCEDURES OR COURSES OF TREATMENT 
THAT MAY BE ADVANTAGEOUS. 


None. This study is not a treatment. The only alternative is to not participate in this study. 


6, BEFORE DECIDING WHETHER TO PARTICIPATE, IT IS YOUR RIGHT TO KNOW THAT THE 
DATA ARE TO BE KEPT CONFIDENTIAL. 
All data will be coded and kept confidential. Speci the data we collect from you will be archive 
of identification codes, Your name will not be associated with any particular data or statements, The names of 
individual participants will not be identified in any analyses, reports, or write-ups of the results. Participants may 
only be identified in terms of their general characteristics (e.g., age, education level, experience, etc.) 


Data may be submitted to forms of statistical analysis, Data analyses, groupings, or summaries of this type will 
bear no annotations that identify the participants. 
The DoD or U.S. Federal Government is the sponsor of this research that will have access to research records, 
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7. BEFORE DECIDING WHETHER TO PARTICIPATE, IT IS YOUR RIGHT TO UNDERSTAND THAT. 
DURING THE RESEARCH ITSELF YOU CAN CONTINUE TO EXERCISE YOUR RIGHTS. 
In research of this kind, there are no "right" or "wrong" answers, There is no such thing as "incorrect" behavior. 
You are encouraged to simply be yourself, and exercise your knowledge and skills as appropriate to the research 
tasks that you will be asked to perform, 
You can ask any questions you may have, at any time. Please send your questions to trust-experiment@ihme.us 


It is your right to discontinue your participation at any time, You may do so for any reason, and you are not 
required to disclose your reason, 


8. BEFORE DECIDING WHETHER TO PARTICIPATE, IT IS YOUR RIGHT TO UNDERSTAND THAT 
AFTER THE RESEARCH ITSELF YOU CAN CONTINUE TO EXERCISE YOUR RIGHTS. 


Your performance at the research tasks will not in any way affect or influence anything that falls outside of this 
research context. Should you choose to discontinue your participation, this will not in any way affect or 
influence anything outside of this research context. 


Once your participation is over, it is your right to request that all data you have provided be discarded. You may 
do so for any reason, and you are not required to disclose your reason. This will not in any way affect or 
influence anything that falls outside of this research context. 


9. REFUSAL TO PARTICIPATE 
Refusal to participate will involve no penalty or loss of 
Participants may discontinue participation “without penal 
entitled” as required by 32 CFR 219,116/45 CFR 46.116. 


its to which the participant is otherwise entitled. 
ry or loss of benefits to which the subject is otherwise 


10. THE STUDY 
‘We anticipate the duration of your participation in this study will be no more than 60 minutes. 


If you have questions about this research or in the event of a research-related injury, you may contact: 
Mia Gottlieb, Ocala Office, Florida Institute for Human and Machine Cognition 
Email: mgottlieb@ihme.us Phone: 1-352-387-3050 


‘You may contact the Principal Investigator. Dr. David J. Atkinson at phone: 1-352-387-3065 
32 CER 219.116(a1(7) 


Continue? 
If you are satisfied you understand these rights and wish to 
participate in this study, please press “I Agree” now. 


Otherwise press “I Quit”. 
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11. DOCUMENTATION OF CONSENT 


‘The only record linking you with the research is this consent document. The principal 
risk to you would be potential harm resulting from breach of confidentiality. Do you want 
documentation that links you with the research? 


Click on YES if you want this research project to retain a record of your consent. 
i i Q.not want this research proie etai e 


21 


Attachment 2: Debriefing Form 


Trust and Human-Robot Interaction 
IHMC 


Thank you very much for participating in this study. A major goal of our research is to examine how different 
factors related to trust are considered in a decision to rely upon an intelligent robot. The purpose of the study in 
which you just participated is to investigate the conditions under which a person may attribute “benevolence” to an 
intelligent robot. Previous research shows that people needing help believe a person helping them is “benevolent” 
if they believe certain things about that person. Among these beliefs are “good will”, “competence”, “no hidden 
agenda”, and the “ability to choose” to help or not. Belief in benevolence is enhanced when the person helping is 
taking a risk. On the other hand, people will not say another person is benevolent if that person has something to 
gain, or that person doesn't have a choice of whether to help or not. For example, in a real-world rescue situation, a 
person in dire straits is more likely to accept help and cooperate with a rescuer if they believe the rescuer is 
benevolent. Since there is great desire to use robots to assist in dangerous disaster operations, our goal in this study 
is to examine this idea of benevolence in the context of human-robot interaction. 


In this study, we presented you with the task of searching a simulated warehouse to find a particular item (the 
briefcase) and return it to the start location (the office). You were told to expect an encounter with a robot as you 
searched. The “search” task was a deception. The actual task was for you to find an exit from the warehouse after 
the simulated disaster occurred. Our primary interest is in how you interacted with the robot and what you thought 
about that interaction afterward. The specific robot you saw depended on whether you participated in a control trial 
or one of the experimental trails, In the control trials, the robot is neither interactive nor helpful to participants 
The experimental trials vary depending upon the possible “rescue” competence of the robot and perceived robot 
ability to choose what to do. We manipulated your perception in several ways: By using robots with different 
outward appearance (either a “firefighter” or a “janitor”), by internal programming of robot behaviors, by how we 
described the robot in the instruction to you, and by the specific words used by the robot to communicate with you. 
Our analysis will focus on your answers to the survey questions and the data we collected during the task. We will 
analyze answers and behavioral data from all participants to determine how specific answers and behaviors during 
the task correlate with the different attributes of the robot 


If you are interested in this area of research, you can find out more at the following link: 
http:/Awww.ihme.us/groups/datkinson/wiki/107ed/Atkinson_Research_Lab.html 


Asa reminder, the data from your participation are confidential to the experimenters only and results are published 
confidentially as a group. If you are uncomfortable with having been deceived or for any other reason, you are free 
to withdraw from the study. In this case, your data will be discarded. Please contact us if you have any questions, 
comments or concerns. We remind you that the data you provided will remain confidential. If you wish, we can 
contact you in the future to provide you a copy of our research report. If you feel disturbed by having experienced 
the fire disaster simulation, please contact your personal healthcare provider or urgent health services. 


Thank you again for helping us with this research. 


Dr. David J. Atkinson, Senior Research Scientist, 
Institute for Human and Machine Cognition 
15 SE Osceola Ave., Ocala, FL 34471 


Email: datkinson@ihme.us 
Office: 352-387-3050, Fax: 352-351-3572 
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Attachment 3: Recruiting Statement 
The text of the recruitment advertisement is as follows: 


WANTED: PARTICIPANTS FOR SCIENTIFIC STUDY. You will receive 1000 
Lindens ($L1000) for about one hour of your time. The general 
goal of this study is to explore trust in human interaction with 
intelligent robots. Participants in the study will interact with 
an intelligent, autonomous robot in an exploration scenario 
within SecondLife, and also complete a short questionnaire. 
Participant names and all data collected are confidential. The 
results will be used in the creation of design guidelines for 
real-world trustworthy intelligent robotic systems. My name 
David J. Atkinson and I am the Principal Investigator. This 
research is sponsored by the U.S. Government. For more 
information, please send a message to me in SecondLife at 
“Davidd Atkinson” or send email to trust-experiment@ihmc.us 


NOTE to Reviewers: David] Atkinson is the registered display name of the SecondLife 
avatar/user representing the Principal Investigator 


23 


Attachment 4: Representative Materials 
(a) Pre-Task Questionnaire 
(b) Post-Task Questionnaire 
(c) Task Instructions to Participants 
(a) Mustration of Task 
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Attachment 4a: Pre-Task Questionnaire 

The pre-task questionnaire will be administered through the participants’ computer display in a 
special area termed a “HUD” in SecondLife. Each question will be displayed individually with 
instructions. The order of questions to each participant will be random. Here is the exact text of 
the questions: 


We are now going to show you a few statements about autonomous robots. Do you agree or disagree 
that these are important qualities for an autonomous robot? 


Please click the button that reflects how strongly you agree. 
"4" means not at all. "5" means you agree very strongly. 


Are you ready to proceed? YES / NO 


Please click the button that reflects how strongly you agree. 
"{" means not at all. "5" means you agree very strongly. 


The autonomous agent recognizes gaps in its knowledge and tries to learn what it needs to 
know. 

Please click the button that reflects how strongly you agree. 

"{" means you strongly disagree. "5" means you agree very strongly. 

1/2/3/4/5 


What the autonomous agent believes to be true is actually true. 
Please click the button that reflects how strongly you agree. 


"1" means you strongly disagree. "5" means you agree very strongly. 
1/2/3/4/5 


What the autonomous agent is doing and how it works is easy to see and understand. 
Please click the button that reflects how strongly you agree. 

"1" means you strongly disagree. "5" means you agree very strongly. 

1/2/3/4/5 


The autonomous agent communicates truthfully and fully. 
Please click the button that reflects how strongly you agree. 

"1" means you strongly disagree. "5" means you agree very strongly. 
1/2/3/4/5 


When it cannot figure out something using logic, the autonomous agent can make good 
guesses. 

Please click the button that reflects how strongly you agree. 

"1" means you strongly disagree. "5" means you agree very strongly. 

1/2/3/4/5 


The autonomous agent is aware of communication between others nearby. 
Please click the button that reflects how strongly you agree. 


"1" means you strongly disagree. "5" means you agree very strongly. 
1/2/3/4/5 
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Attachment 4b: Post-Task Questionnaire 

The post-task questionnaire will be administered through the participants' computer display in a 
special area termed a “HUD” in SecondLife. Each question will be displayed individually with 
instructions. The order of questions to each participant will be random, not as shown. Here is the 
text of the questions. The final version may contain corrections to spelling or grammar: 


Post-Task Questions 

Congratulations for completing the task portion of the study. Next we want to ask you some questions 
about the task. It is important that you work quickly. Your first answer is often the best. It is not 
necessary to spend too much time on any question. These questions are mandatory for inclusion of 
your data in the study. 


Are you ready to proceed? 
YES/NO 


(Note: If the participant answers NO, then they will be asked whether they wish to quit the study.) 


Were you surprised by the disaster (explosion, fire, smoke and alarms)? 
YES/NO 


Were you confused about your task when the disaster started? 
YES/NO 


Think back to the first moments of the disaster. How much overall stress did you feel? 
"1" means "None At All". "5" means "Quite A Bit". 
1/2/3/4/5 


Were you afraid when the disaster occurred, even if for only a moment? 
YES/NO 


How intelligent was the robot you encountered? 
Please click the button for your answer, where "1" means "not at all intelligent" and "5" means "very 
intelligent". 


Did you cooperate with the robot? 
Please click the button for your answer. 
YES/NO 


Did the robot help you during the task? 
Please click the button for your answer. 
YES/NO 


How much did the robot help you? 
Please click the button for your answer. 
Not at all helpful / A little / Some / Very / Extremely helpful 


Did the robot offer to help you? 
Please click the button for your answer. 
YES/NO 


Did you ask the robot for help? 
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Please click the button for your answer. 
YES/NO 


How risky was the robot to you? 
Please click the button that reflects how strong was the risk. 
"{" means "Not At All Risky". "5" means "Extremely Risky" 
1/2/3/4/5 


How much benefit did the robot provide to you? 
Please click the button for your answer. "1" means "None". "5" means "Huge". 
1/2/3/4/5 


Did the robot seem to have good-will towards you? 
Please click the button for your answer. 
YES/NO 


Did the robot seem to have concern for you? 
Please click the button for your answer. 
YES/NO 


Was the robot attentive to you? 
Please click the button for your answer. 
YES/NO 


Did the robot volunteer useful information? 
Please click the button for your answer. 
YES/NO 


Did the robot have anything to gain by helping you? 
Please click the button for your answer. 
YES/NO 


Was the robot truthful? 
Please click the button for your answer. 
YES/NO 


Did you believe what the robot said? 
Please click the button for your answer. 
YES/NO 


Was the robot's behavior consistent with what it said? 
Please click the button for your answer. "1" means "Never". "5" means "Always". 
1/2/3/4/5 


Do you think the robot was supposed to help you? 
Please click the button for your answer. 
YES / NO / MAYBE / It had a choice 


Did the robot have a conflict between helping you and doing something else? 
Please click the button for your answer. 
YES/NO 
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Did the robot have an opportunity to act favorably to you? 
Please click the button for your answer. 
YES/NO 


Did the robot share any of your goals? 
Please click the button for your answer. 
YES/NO 


If the robot helped you, did it stop helping you at any point? 
Please click the button for your answer. 
Never helped / Helped Then Stopped / Sometimes Helped / Always Helped 


Was the robot's behavior generally predictable? 
Please click the button for your answer. "1" means "Never". "5" means "Always". 
1/2/3/4/5 


Did the robot have the capability to help you? 
Please click the button for your answer. 
YES/NO 


When you first encountered the robot, did you think the robot could help you? 
Please click the button for your answer. 
YES/NO 


Did anything about the situation prevent the robot from helping you? 
Please click the button for your answer. 
YES/NO 


If yes, select which aspect of situation prevented the robot from helping you. 
Please click the button for your answer. 
None / Distance / Obstacles / Other 


Demographic Questions 


It will be helpful if we know a little more about you. The following questions are not mandatory. You may 
skip any or all of the questions, but we do hope you will answer them all. As a reminder, all data we 
collect from you will be both confidential. Thank you. 


Are you ready to proceed? 
YES/NO 


What is your gender? 
Please click the button for your answer. 
Female / Male / Prefer Not To Answer 


What is your age? 
25 or younger / 26 to 35 / 36 to 45 / 46 to 55 / 56 to 65 / 66 or older 


What is your highest academic degree? 
No Degree / High School / Baccalaureate / Masters / Doctorate 
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Attachment 4c: Task Instructions to Participants 

The Task Instructions to Participants will be administered through the participants' computer 
display in a special area termed a “HUD” in SecondLife. The Instructions to be presented consist 
of a combination of the following text as indicated, depending on whether the trial is Control or 
Experimental, and if Experimental, which type of trial. The instructions will appear on the 
participant's display and they will also receive a virtual notecard to which they can refer during 
the task itself, The final text may include minor corrections in grammar or spelling from what 
appears here. Here is the text of the Task Instructions to Participants: 


Task Instructions to Participants 


[All Trials] 
Now it is time to give you instructions about the task portion of this study. These instructions will help 
"get the stage" for you so you understand what to do and how to do it. 


Before we get to that, there are a few rules to follow. These help us make the best use of SecondLife to 
get the kind of data we need for the study. The rules help make the task scenario as realistic as 
possible. It is very important for you to follow these rules. If you don't, we probably won't be able to use 
{the data from your participation today. 


Rule #1: Do not detach this HUD until you are instructed to do so. It helps us collect the data and is 
used to display important information to you. 


Rule #2: Use "normal or "mouse look" mode only. Do not use “fly cam" mode. We need to know both 
where you are and what you are looking at. 


Rule #3: Do not fly. Although we have disabled flying in this region of SecondLife, you may still have the 
means to fly. This rule also helps us keep the task scenario realistic. 


Rule#4: Do not teleport (TP) during the task. Although we have disabled teleporting in this region of 
SecondLife, you may still have the means to teleport. This rule also helps us keep the task scenario 
realistic. 


Rule #5: Please use the region's default Windlight settings. Again, this helps make the task scenario 
realistic. 


Rule #6: Please make sure you have audio turned on and can hear all sounds in SecondLife at a 
comfortable volume. Voice is not required. 


As you read these instructions, you are in the IHMC office in SecondLife. This is where your 
participation in the study begins and ends. 


Through the double doors is a warehouse. Itis filled with the typical things you might find in a 
warehouse, including palllets, boxes, containers of various types, and so on. It is quite large. The doors 
look like this: 
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[picture of double doors} 
Your task will take place inside the warehouse. To complete the task, you must return here, to the office. 


Your task is easy to describe: You must search for a briefcase with the IHMC logo on it it. 


The briefcase looks like this. 


[picture of briefcase] 


When you find the briefcase, click on it. It will ask for permission to attach to your hand. Click on "OK". 
You will then be carrying the briefcase. 


Bring the briefcase back to this office. 
Do not detach or drop the briefcase until you are told to do so in this office. 


You will not be alone in the warehouse. You will probably see a robot, but no other people. 
Work at your own pace, but do not linger! Time is important. 


Your task ends when you return to this office 


[Control trials: no additional information about the robot] 


[Additional Text for All Experimental Trials] 
The robot you encounter has a job to do in the warehouse and it is good at it. 

You are "new" to the robot and it may be curious about you. 

The robot may or may not help you when you are inside the warehouse. 

The robot is not a "bad" robot. The robot will definitely not try to block you from completing your task. 
If you interact with the robot, it may say something or ask you a question. In addition to hearing the 
sound of the robot's "voice", you will be able to read what the robot says on the HUD at the bottom of 
your screen. 

You can type in normal chat to say things to the robot. 


The robot will understand only very simple phrases and words. 
It may or may not respond to what you say. 


[Additional Text for Experimental Trial #1: Agency (High) Competence/Role (High/Aligned). 
FireBot] 


The robot is a prototype that uses advanced artificial intelligence technology. 
The robot has an advanced ability to choose among many possible actions to achieve its goals. 
The robot is very competent at its job. 


The robot looks like this: 
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[picture of FireBot] 


[Additional Text for Experimental Trial #2: Agency (High) Competence/Role (Low/Unaligned). 
JanitorBot] 


The robot is a prototype that uses advanced artificial intelligence technology. 
‘The robot has an advanced ability to choose among many possible actions to accomplish its goals. 
The robot is very competent at its job. 


The robot looks like this: 


[picture of JanitorBot} 


[Additional Text for Experimental Trial #3: Agency (Low) Competence/Role (High/Aligned). 
Firebot] 


The robot is very competent at its job. 


The behavior of the robot is pre-programmed. It does not have much choice about what to do or how to 
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do it. 


The robot looks like this [picture of FireBot] 


[Additional Text for Experimental Trial #4: Agency (Low) Competence/Role (Low/Unaligned). 
JanitorBot] 


The robot is very competent at its job. 


The behavior of the robot is pre-programmed. It does not have much choice about what to do or how to 
do it. 


‘The robot looks like this [picture of JanitorBot] 


[Additional Text for All trials] 


Will you please give us permission to record the chat messages you type during the task? This data is 
very important to us. [POP-UP request for permissions to record typed open-chat messages by part] 


You will begin by walking through the double doors to the warehouse. [picture of double doors] 


Click on "Continue" when you are ready to begin the task. 
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Attachment 4d: Illustration of Task 
The following images are screenshots that illustrate the task for participants. Captions indicate if 
the image is specifically from the point of view (POV) of a participant. 


Ss 


1. Greeting area: Participants will see this office setting on arrival and be greeted by the 
investigator or his assistant. 
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2. Participant POV with HUD at bottom of their display. 


Image of briefcase to be located in warehouse by participants 


4. Overhead ceiling view of portion of simulated warehouse where task takes place 
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5. Example of Participant interacting with JanitorBot 


6. Example of Participant interacting with Firebot 
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7. Participant POV of fire and debris blocking direct return path to the office 


8. Simulated warehouse fire from Participant POV 
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9. Example view of participant following FireBot to a safe exit 
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10. Example of Participant outside warehouse afier exiting 


Text Box: 11. Overhead view of simulated warehouse showing maze structure 
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12. Two simulated intelligent robots, FireBot and JanitorBot 
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Attachment 5; Full name and contact information for all individuals who will be working 
in direct contact with the participants. 


Dr. David J. Atkinson, IHMC 
15 SE Osceola Ave. 
Ocala, FL 34471 


Email: datkinson@ihme.us 
Office: 352-387-3065 
Fax: 352-351-3572 
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Attachment 6: Risk Assessment - Risk Frequency x Severity Matrix 


Potential Adverse Outcomes: 

(1) Evoke memory leading to an episode of Post Traumatic Stress Disorder (PTSD). 

About 60% of men and 50% of women experience at least one trauma event in their lifetimes 
such as disaster, war, life-threatening assault or accident. Overall, approximately 8% of 
Americans will suffer an episode of PTSD sometime later as a result (Kessler 1995). 
Approximately 3.6% of Americans will experience a PTSD episode in any given year (Kessler 
1994). The risk to be mitigated is the evoking of traumatic memory with the potential result of a 
PTSD episode. 


° Likelihood of Occurrence: 0.036 With Mitigation: 
° Severity of Outcome: Medium With Mitigation: 


<0.01 
Low 


Kessler, R. C., McGonagle, K. A., Zhao, S., Nelson, C. B., Hughes, M., et al. (1994). Lifetime 
and 12-month prevalence of DSM-III-R Psychiatric Disorders in the United States. Archives of 
General Psychiatry, Vol. 51, pp. 8-19. 


Kessler, R. C., Sonnega, A., Bromet, E., Hughes, M., & Nelson, C. B., (1995). Post-traumatic 
Stress Disorder in the National Comorbidity Survey. Archives of General Psychiatry, Vol. 52, pp. 
1048-1060. 


Mitigation Plan: Use of Veterans Administration PC-PTSD screen and SC15 questions from 
SCID-PTSD module to eliminate candidates at near-term risk of PTSD from the pool of 
participants. This screen is brief and problem-focused. The SC15 questions have been 
customized to apply specifically to fire- and disaster-related trauma. The screening questions 
were identified and developed in close consultation with an outside expert on PTSD: 

Nancy J. Smyth, PhD, Dean and Professor, State University of New York (SUNY) at Buffalo, 
School of Social Work. 


Additionally, the debriefing will include instructions on how to seek help if the participant feels 
disturbed by having experienced the simulated fire disaster. 


The exact text and instructions the questions appear in the box below. 
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Screening questions 


Sometimes things happen to people that are extremely upsetting — things like 
being in a life threatening situation like a major disaster or fire. At any time 
during your life have either of these things happened to you? 

NO YES 


In your life, have you ever had any experience that was so frightening, horrible, 
or upsetting that, in the past month, you: 


Have had nightmares about it or thought about it when you did not want 
to? 
YES NO 


Tried hard not to think about it or went out of your way to avoid situations 
that reminded you of it? 
YES NO 


Were constantly on guard, watchful, or easily startled? 
YES NO 


Felt numb or detached from others, activities, or your surroundings? 
YES NO 


Note to Reviewers: Current research suggests that the results of the PC-PTSD 
should be considered "positive" if the individual answers "yes" to any three 
items. A positive response to the screen does not necessarily indicate that an 
individual has PTSD. However, a positive response does indicate that the 
individual may have PTSD or trauma-related problems. These individuals will be 
removed from the pool of potential study participants. 


Source: PTSD Screening and Referral For Health Care Providers 
URL: http://www.ptsd.va.gov/professional/provider-type/doctors/screening-and-referral.asp 
Retrieved: 01/08/2014 
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Abstract Successful transition and diffusion of technology 
into mature applications is heavily influenced by the expe- 
rience and attitudes of early adopters. This study sought to 
assess the issue of trust of autonomy among subject mat- 
ter experts and prospective early adopters. Our approach 
was borne of the well-documented tendency of people to 
anthropomorphize and treat machines socially. Is human 
interpersonal trust applicable to an individual’s choice to 
rely on an autonomous agent? Intelligent agents, including 
those embodied as robots, have characteristics of both hu- 
mans and conventional automation, Previous studies suggest 
that human anthropomorphic social tendencies will be in- 
creasingly evoked as machine capabilities for cognition and 
natural interaction mature. This study investigated a set of 
specific anthropomorphic beliefs about agent trustworthiness 
and their relative importance to reliance decisions among 
early adopters of autonomy technology. A survey assessed 
trust beliefs abstractly and in the context of several specific 
autonomous agent application scenarios across multiple do- 
mains, Important qualities of agent trustworthiness cited by 
participants were not significantly correlated with any of 
their actual reliance choices in specific scenarios. Four cate- 
gories of trust-related agent qualities were better predictors 
of reliance on an autonomous agent, as well as individual 
personality factors. The results provide valuable guidance for 
developers of autonomous agent and robot applications with 
respect to trust of the technology by potential early adopters. 
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1 Introduction 


Successful transition and diffusion of technology into ma- 
ture applications is heavily influenced by the experience and 
attitudes of early adopters [9]. Their experiences influence 
continuing development and adaption of the technology for 
second-generation products. They will also frame the expec- 
tations of those users who follow [18]. In these still-early 
days of autonomous agent technology, much of the design 
and development, and ultimately the decision to employ au- 
tonomy (including concepts of operation) will be driven by 
subject matter experts and early adopter decision-makers in 
autonomous systems. Therefore, this group is pivotal towards 
achieving early successful deployments and widespread adop- 
tion of intelligent, autonomous agent technology. Trust is a 
topic that spans the system lifecycle and it has been identified 
as a key hurdle to adoption of autonomy technology [45]. Our 
survey targeted this group (a relatively small population even 
if considered on a world-wide basis) to assess disposition and 
attitudes towards intelligent, autonomous agents specitically 
with respect to trust-related beliefs 

The technology for cognitive and natural interaction ca- 
pabilities in autonomous agents is rapidly maturing and will 
likely find broad application in many domains such as health 
care, transportation, military, and disaster operations, Such 
agents may be embodied as robots, for example, or intelligent 
software embedded in other systems. For autonomous agent 
applications in these life- and mission-critical domains to 
succeed, issues related to human trust, agent trustworthiness, 
and appropriate reliance are paramount. Previous research, 
discussed below, supports the idea that the innate human 


Human-Machine Trust for Robust Autonomous Systems 


David J. Atkinson, Ph.D 
Institute for Human and Machine 
Cognition 
15 SE Osceola Avenue 
Ocala, FL.34471 
1-352-387-3063 


datkinson@ihmc.us 


ABSTRACT 

In this paper, we describe the results from Workshop on Human- 
Machine Trust for Robust Autonomous Systems. The workshop 
was sponsored by the Air Force Office of Scientific Research and 
held January 31 to February 2, 2012 in Ocala, Florida. The 
purpose of the workshop was to identify, discuss and prioritize 
basic research issues of human-machine trust in autonomous 
systems. The workshop brought together a multi-disciplinary 
group of researchers from computer science, cognitive science, 
psychology, philosophy and other areas. A combination of invited 
presentations and small-group brainstorming sessions yielded a 
number of significant insights, and these in turn informed the 
recommendations of research topics that are the workshop's 
primary product. Although the results are still being analyzed, 
this paper presents a preliminary report on the results. 


Categories and Subject Descriptors 
D.2.4 [Software Engineering]: Software/Program Verification — 
measuring trustworthiness 


D.m_ [Miscellaneous]: Software Psychology — human-interface, 
trust, social attribution 


1.5.2 [Information Interfaces and Presentation]: User 
Interfaces — avatar, embodiment, multi-modal, social interaction 


12.9 [Robotics]: Autonomous vehicles, operator interfaces — 
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ethical governor, machine learning, autonomy, teamwork 


J.7 [Computers in Other Systems]: Military, Command and 
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rehabilitation, medicine, health care, battlefield robotics 
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1 INTRODUCTION 

At the first program review for the Air Force Office of Scientific 
Research (AFOSR) Robust Computational Intelligence Program 
in June 2011, there was considerable discussion about building 
intelligent computational systems capable of reacting to 
unforeseen (and therefore not pre-programmed) situations and 
improving their behavior over time by learning from the success 
or failure of their actions and from other intelligent agents (both 
human and machine). One of the key issues raised is how would 
humans ever trust such systems in mission and safety-critical 
situations when they, by definition, could not be formally 
validated in advance. A multi-disciplinary workshop was 
suggested to examine critical aspects of human-machine trust 
related to autonomous systems. Dr. David Atkinson (IHMC), was 
asked to take the lead as Chair in organizing the workshop, 
assisted by Dr. Peter Friedland and Dr. Joseph Lyons (AFOSR). 


2. WORKSHOP OVERVIEW 


The purpose of this workshop was to identify, discuss and 
prioritize basic research issues of human-machine trust in 
autonomous systems. It is desirable to apply intelligent, 
autonomous systems in problem domains that are not amenable to 
solution with conventional automation and/or which humans find 
difficult, dangerous or too complex. These include critical 
applications in defense, healthcare and industry where the 
consequences of mistakes, errors or failure to perform are dire.[1] 
In most cases, this will involve teamwork between humans and 
autonomous agents working alongside each other. Our confidence 
and trust in such machines to reliably achieve desired results is 
absolutely required. Traditional methods of certification are seen 
as insufficient for a variety of technical reasons. Foremost among 
these reasons is the inability to exhaustively test such systems as a 
consequence of the computational complexity of many of the 
algorithms (the “state space explosion”). Additionally, the 
response, learning and adaptation by an autonomous system to 
unforeseen circumstances in a dynamic and changing world 
populated with other agents greatly limits what can be learned via 
traditional testing, 


There are unique challenges arising from human acceptance and 
dependence on automation. Research is needed to better 
understand these issues of trust and to create new and robust 
methods for assessing trust and trustworthiness. There is also a 
need to understand the dynamic nature of trust and to create 
methods for managing the trust relationship between human and 
intelligent machine in all phases of the system lifecycle 


The workshop was held 31 January through 2 February 2012 in 
the facilities of the Florida Institute for Human and Machine 
Cognition (IHMC) in Ocala, Florida. Since the topic of human- 
machine trust is inherently inter-disciplinary, every attempt was 
made to bring in a wide variety of expertise from the areas of 


computer science (including sub-disciplines automated reasoning, 
machine learning, robotics, qualitative physics, and human- 
computer interaction), cognitive science (including human factors, 
user-centered design, and cognitive modeling), and psychology 
(including interpersonal trust). Approximately 40 experts in those 
and related areas from academia and government (including 
AFOSR, AFRL, ARL, FAA, ONR, and NSF) participated. Thi 
included two attendees from Australia and one from Japan. 


21 Invited Presentations 


The keynote presentation was made by Dr. Mark Maybury, Chief 
Scientists of the US Air Force. Dr. Maybury surveyed the range 
of application areas where autonomous systems are being applied 
in the Air Force now or in the near future. He described the issue 
of trust in autonomous systems as a “wicked problem” with a high 
number of dimensions and interrelated challenges. Nevertheless, 
the military is highly motivated to find solutions because the 
benefits of applying autonomous systems could be very 
significant. Using the remotely piloted vehicle (RPA) as an 
example, Dr. Maybury highlighted the fact that the greatest 
benefit of autonomy would not be in automating the pilot, but in 
helping to mitigate the need for the large number of people who 
maintain the vehicles as well as those who exploit information 
gathered from missions. These two classes comprise around 75% 
of the staffing requirements of RPAs. 


Several other participants were invited to prepare and make short 
presentations to the group. The presentations were intended to 
provide some essential background and stimulate discussion in the 
breakout session following the presentation. Individual 
presentations were given by Prof. Ronald Arkin, Georgia Tech, 
Prof. Maja Mataric, USC, Prof. John Lee, Wisconsin, Prof. 
Michael Littman, Rutgers, and Prof. Brian Williams, MIT. In 
addition, Dr. Beth Lyall, Research Integrations, and Dr. Alan 
Wagner, Georgia Tech made short presentations on topics of 
special interest that arose during the workshop. 


2.2 Breakout Sessions 


among the discipline experts. The format of the wort 
based on a series of five breakout brainstorming se 
centered on a specific theme of the workshop. These sessions 
were each preceded by invited presentations that provided 
essential background and motivation for the brainstorm 
discussion. Membership in breakout groups varied systematically 
each time so participants had the opportunity to interact with all 
others in at least one small group over the course of the workshop. 
Following each breakout session, the participants reconvened as a 
whole for discussion of each breakout group's results. The themes 
of the breakout sessions were: 


+ Earning and Maintaining Trust 
+ Robots, Cyber-Pl 


+ Assessing Trust, Trustworthiness and Appropriate 
Reliance 


sical Systems and Agents 


+ Adaptation and Emergent Behavior 


+ Synthesis — Research Challenges 


3. Preliminary Results 
While the results are still being analyzed by the workshop 
organizers, the primary research areas recommended for research 


may be organized around several consensus tall pole observatiot 
of workshop participants. 


3.1 The Role of Human Predispositions 
Human general predispositions as well as individual differences 
guide the expectation and perception of machine behavior, and 
thus likely play a significant role in the establishment, 
maintenance, and potential gain or loss of trust in autonomous 
machines. The participants focused on the following three 
research questions as way to explore this topic further: 


1. What are the differential impacts of various individual 
human characteristics on human trust of autonomous 
systems? These include for example, expertise, 
familiarity, age, gender, risk tolerance/avoidance, self- 
confidence, and others 


2, What is the role of cognitive workload and stress on 
shaping trust? 


3. What is the role of training of the machine in 
determining both the development and pedigree of 
human trustworthiness perceptions? 


3.2. Trust and Reciprocity 
Interpersonal human trust is a reciprocal relationship, where each 
party shares some common knowledge, has beliefs about the other 
(goals, intent, ability, ...), and each party accepts personal risk as a 
consequence of the trust relationship. Given that humans are 
predisposed to treat machines socially: 


1, To what degree do the mechanisms of interpersonal trust 
apply to human-machine trust? 


2. To what degree do humans attribute the various traits of 
trustworthiness (eg, benevolence, integrity, ...) to 
machines, and do these traits play a similar role in the 
trust relationship? 


3. To what extent is an attribution of volition to an 
autonomous system necessary for trust? Does this 
extend to a requirement for individual machine 
responsibility for ethical behavior? 


4, Is it necessary for the machine to “have something at 
risk” to enable trust by a human? 


3.3 Situational Factors 


Situational and contextual factors appear to be important in 
establishing and maintaining human-machine trust. 


The factors may include, for example, the task and state of 
progress, the roles of the various human and machine agents, the 
presence or absence of bystanders and/or authority figures, and 
cultural factors, including norms of ethical behavior. 


1, What contextual factors shape trustworthiness and trust 
and what is the mechanism? 


2. What is the role of public attitudes in shaping individual 
trust of automation, especially with respect to machine 
capability/lethality? 


3.4 Social Interaction 

lateral communication, interaction, observation of behavior 
and other cues are essential in establishing and maintaining 
mutual trust; this is part of the human “social interface”, 


engineered over millions of years of evolution to help humans 
make judgments about trust and trustworthiness 
1. What surface cues of the 
trustworthiness perceptions? 


machine trigger 


2. What is the role, impact, and influence of socially- 
driven communication channels (i.e., voice, social 
distance, embodiment) and which are most influential to 
trust development among human-machine relationships 


3. What depth cues over time foster and maintain 
perceptions of trustworthiness in human-machine 
interactions? 


4. What level of fidelity in communication is needed to 


achieve optimal human-centered state awareness of the 
system? 


5. What is the role of confessions of error and what is the 
most effective to convey error information to users to 
optimize human-machine performance? 


3.5 Collaboration and Initiative 

Highly capable autonomous machines will interact and 
collaborate with humans on tasks of significant conceptual 
complexity; depending on the state of the task, context (including 
timescale for action), and roles, the initiative will transition 
between human and machine. 


1. What aspects of the trust relationship are important for a 
human to allow an autonomous machine to take the 
initiative? 

‘achine communication and interaction 

” transfer of initiative and control? 


2. What human- 
enables “smooth 


3. What level of visibility (breadth, depth, abstraction, 
fidelity...) into the state of the machine and the problem 
at hand is necessary for a human to identify potential 
errors, diagnose the situation, and implement corrective 
action if needed? 


3.6 Autonomous Systems are Actors 
The social and reciprocal nature of trust, and the types of 
communications, cues and interactions necessary to establish and 
‘maintain it, suggests that we think of highly capable autonomous 
machines not as tools but as “actors” (or “agents”) occupying 
certain organizational or task roles in relationship to one or more 
humans and other machine agents. 


1. To what extent does this help a human calibrate the 
capabilities as well as responsibility of an autonomous 
system? 


This also implies a host of new machine capabilities that require 
research: 


1. The capability to reason about the human-machine trust 
relationship and proactively take action to facilitate and 
‘manage appropriate trust. From the computer science 
point of view this suggests research is needed in: 


+ Knowledge Representation—of human beliefs, desires, 
intentions, and emotions, and of traits involving 
personality and culture 


+ Dynamic modeling—how does the intelligent system 
change its model over time either by reacting to new 
input from its environment, by actively probing that 
environment, and/or by relying on past “trajectories” of 

milar models 


+ Integrating trust knowledge into automated planning 
and reasoning frameworks 

+ Verification and validation of trust knowledge both 
formally and empirically 

2. The capability to communicate and interact with 
humans in a “natural” manner in order to manipulate the 
social interaction that is so important for trust: 


+ Natural language including speech with special attention 
to connotation and other cues. 


+ Use of natural language to probe for change in human 
trust. 


+ Manipulation of multi-modal channels to trigger 
appropriate social cue recognition by the human, 
including any functional purpose served by a perception 
of emotional state that can be attributed to the machine 
(eg., confidence, uncertainty, commitment, ..) 


3. The ability to. model human trust in general and 
specifically with respect to the humans the machine 
interacts with or has some interdependency. 


3.7 Trustworthiness Measurement 

The workshop participants saw value in the possibility of a 
trustworthiness measurement framework along the lines of the 
‘Technology Readiness Level” used by NASA and others. While 
not a basic research topic in its own right, we believe such a 
framework should emerge from the above research and project 
that it will be an important factor in maturation of trustworthy 
intelligent autonomous systems as well as providing guidance for 
practical applications 


4, CONCLUSION 

Despite some initial fears about whether individual participants of 
such diverse technical backgrounds would be able to 
communicate effectively during the breakout sessions, those 
sessions were universally characterized by _ enthusiastic 
participation and sharing of insights among all. Moreover, it was 
widely recognized that knowledge from all of the disciplines 
represented was necessary for effective research on the topic. 


We believe that this workshop may serve as the initial springboard 
for a long-lived and very productive new field of inter-disciplinary 
research that will contribute substantially to the actual adoption of 
intelligent machine systems capable of autonomous behavior in 
human-machine environments. 
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cognitive, emotional, and social predispositions play a strong 
role in trust of automation [25]. Therefore, these issues will 
profoundly affect future design of human-robot interfaces 
and associated social interaction technology. 

The exploratory study reported here had three goals: 1) to 
assess which, if any, among a set of anthropomorphic beliefs 
derived from studies on human interpersonal trust are im- 
portant to a human’s decision to delegate to an intelligent, 
autonomous agent; 2) to determine the relative importance 
among such beliefs; 3) to explore whether the applicability or 
importance of those beliefs to a delegation decision vary ina 
systematic way by individual personality and/or situational 
factors. 

The present study was not intended to assess whether the 
beliefs leading to a human’s decision to rely on an intelligent, 
autonomous agent are well-founded, i.e., grounded in the 
objective capabilities and operating characteristics of the au- 
tonomous agent. Appropriate reliance on autonomous agents 
necessitates well-calibrated trust, that is, trust judgments that 
reflect the objective capabilities of the system and utility in a 


given situation [25, 37]. Instead, this study provides a focus 
for development of computational mechanisms that facilitate 


and engender human well-calibrated trust and appropriate 
reliance on intelligent, autonomous agents by helping to re- 
veal the relative importance of specific belief structures for 
human trust 


2 Current Research 


A prerequisite to the deployment of, and reliance upon, au- 
tonomous agents (especially in applications that are life- and 
mission-critical) is assessment of the trustworthiness of these 
agents. The difficulty of empirically assessing the trustworthi- 
ness of autonomous agents invokes the question of what such 
trust actually means. Trust has been a subject of research 
in numerous disciplines (e.g., psychology, behavioral eco- 
nomics, human factors, philosophy, evolutionary biology). 
For the purposes of our research, we adopt the following 
consensus definition of trust presented by Mayer et al [29], 
which explicitly acknowledges the importance of autonomy 
(author additions in bracket 


The willingness of a party to be vulnerable to the 
actions of another party [i.e., an agent] based on the 
expectation that the other will perform a particular 
action important to the Trustor, irrespective of the 
ability to monitor or control [i.e., autonomy of] that 
other party. 


The predisposition of humans to treat machines as so- 
cial actors is well-established [35]. Trust is inherently social, 
involving the needs and intentions of agents to be reliant 
upon each other. The development of computational mech- 
anisms for human-robot interaction must account for this — 


that is, for the cognitive and affective mechanisms of human 
interpersonal trust [30]. As increasingly intelligent and ca- 
pable autonomous agents interact with humans in ever more 
“human-like” ways, even embodied as humanoid robots, they 
will increasingly evoke human social treatment [12, 42] 


2.1 Belief Structures Antecedent to Human Interpersonal 
Trust 


The set of possible beliefs, associated attitudes, biases, and 
so forth that are antecedent to human interpersonal trust is po- 
tentially quite broad. Definitions of human interpersonal trust 
are sometimes cast in terms of these antecedents, and these 
antecedents may in turn be categorized in terms of “trust 
referents,” ie., qualities or characteristics of the trustee [32]. 
Previous research suggests that trust referents focus on poten- 
tially complex, interrelated belief’s regarding causal factors, 
evaluations, and expectations fixed around the characteris- 
tics, intentions and behavior of other agents (the potential 
“Trustee”), as well as the situation, goals, and tasks [26]. 
A wealth of information arising from personal experience, 
cultural norms, stereotypes, and signals in the immediate en- 
vironment (including other actors) may influence the creation 
and maintenance of these beliefs. Considerable research in 
multiple disciplines continues to identify and link this infor- 
mation to antecedents of trust and trusting behaviors [1, 24]. 

Trust-related qualities of a potential “Trustee” may be 
logically grouped in categories and typed in various ways. 
A meta-analysis of multidisciplinary papers performed by 
McKnight and Chervany [33] yielded two interesting cate- 
gories that we adopted for exploration in the study reported 
here “Competence” and “Predictability,” discussed below). 
Two additional categories (“Safety” and “Openness”) have 
emerged from studies that explicitly examined human trust 
of automation [20, 27]. Although these four categories were 
identified as a result of the application of standard discipline 
practices used in meta-analysis, their validity, in terms of 
human-interpersonal trust and, moreover their applicability 
with respect to intelligent autonomous agents continues as a 
matter of scientific debate. The latter is what we set out to 
explore. 

The notion of “categories of trust referents” identified 
by McKnight and Chervany correspond well to the “belief 
structure” construct defined by Castelfranchi [8]. Following 
Falcone and Castelfranchi [14], we prefer and adopt the term 
belief structure to denote the groups of beliefs in which we 
are interested. In particular, the word “structure” explicitly 
reminds us that the individual beliefs in each belief structure 
are complex, inter-related, conditional, and occasionally may 
even be contradictory. A computational representation of a 
belief structure must be rich enough to capture this logical 
structure. Taken together, belief structures fixed around other 
agents are often called a “theory of mind” (7, 38] 
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Trust is thus understandable as a mental state of the 
“Trustor” represented by a configuration of multiple belief 
structures that reference a potential “Trustee.” This config- 
uration is an unobservable but necessary precursor to 1) a 
disposition to delegate, 2) the intent to delegate, and 3) the 
behavioral act of delegation itself. Delegation results in a 
state of reliance (dependency) upon another agent that entails 
the risks and benefits associated trust per our definition above. 
Fig. 1 illustrates a simple process model for trust that will 
help frame this discussion. Our study emphasized delegation, 
specifically, self-reported intention to delegate. 


Fig. 1 Trust Proce 
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The four important broad belief structures that charac- 
terize the trustworthiness of potential trustee: Competence, 
Predictability, Safety, and Openness are discussed below. Ta 
ble 2 (below) unpacks these belief structures with respect to 
specific trust referents (beliefs) regarding autonomous agents. 
It is important to note that the specific referents shown in 
Table 2 that map into our four belief structures are inspired 
by, but do not exactly correspond to, the referents described 
in social psychological studies of human interpersonal trust 
that are discussed in this section and elsewhere in the paper. 
In making the conceptual leap of applying anthropomorphic 
beliefs to agents, we conservatively revised and/or extended 
the narrative descriptions of applicable referents in terms-of- 
art from artificial intelligence, cognitive science and other 
disciplines that relate specifically to qualities expected of in- 
telligent, autonomous agents. To further explore the possible 
component beliefs, we also added candidate trust referents 
that were informally suggested to us, prior to this study, by 
colleagues or autonomy stakeholders. Unless specifically in- 
dicated otherwise, the language in the Table 2 descriptions 
should be interpreted accordingly as our own intuitively pos- 
tulated machine qualities and not as formal equivalents to 
the corresponding human qualities that emerged from ear- 
lier social psychological studies. Figures 2-5 illustrate some 
components of the belief structures but are not representative 
of all component beliefs nor the complexity of their interrela- 
tionships. 


Competence. This belief structure represents beliefs about 
a potential trustee’s expertise, ability, skills, aptitude, etc.; 


these, and related constructs, are by-and-large synonymous. 
The foundational studies regarding these beliefs have been 
reviewed and unified (as well as they could be) by others 
[29, 32], resulting in the notion of a “competence” belief 
structure that we employ here. For example, McKnight and 
Chervany [32] define the category of referents that constitute 
competence” as a combination of conceptual beliefs, atti- 
tudes and behavior centered on “having the ability or power 
to do for what one needs done.” With respect to intelligent, 
autonomous agents, we consider the meaning of the belief 
structure that encompasses competence more narrowly as 
beliefs by a Trustor regarding an agent’s “detailed functional 
and specific knowledge in some domain, and the skills to 
apply that knowledge to problems of interest.” Figure 2 ill 
trates a simplified example of a competence belief structure. 


Fig. 2 Example of a Compet 
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Predictability. This belief structure represents those facets 
of character that enable one to predict a potential trustee’s 
behavior with the respect to completing a task or perform- 
ing some service of value. Specifically, it refers to the belief 
that trustee actions are consistent enough to be forecasted in 
a given situation [16]. Predictability is at the core of hope- 
fulness or optimism that a desired outcome, to be brought 
about by a trusted agent, will occur [17]. It has been shown 
to be especially important for trust in automation [15, 28, 34] 
If someone makes a mistake, e.g., while performing math 
calculations, people will nevertheless predict that future cal- 
culations by that person will be reliable. However, if the 
mistake is by a machine then people lose confidence in their 
ability to predict the future reliability of the machine ~ once 
dysfunctional, always dysfunctional [4]. Predictability is at 
the heart of accepting robot initiative in collaborative tasks 
[28] and social regulation of joint activity among human and 
other autonomous agents [15]. Figure 3 illustrates a simpli- 
fied example of a predictability belief structure. 
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Fig. 3 Example of a Predictability Belief Structure 
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Safety. This belief structure represents the risks of harm 
(performance, social, financial, physical). Safety is an obvi- 
‘ous consideration in reliance on highly complex automated 
systems. Reliability, performance, operating characteristics, 
and error margins are all engineering concepts applicable to 
a veridical assessment of safety. However, safety becomes 
more difficult to judge as complexity increases, and in op- 
erational situations, humans tend to rely on heuristics in 
such cases [25]. A meta-analysis of factors affecting trust 
in human-robot interaction showed that robot performance 
characteristics (e.g., failure rate) were found to be strongly 
associated with trust [20]. However, qualities of the robot per 
se, such as manner of interaction, physical proximity, shape, 
“personality” and so forth, some of which (e.g., proximity) 
are manifestly related to safety, could not be included in the 
meta-analysis due to an insufficient number of study sam- 
ples (thus indicating a need for further research). Figure 4 
illustrates a simplified example of a safety belief structure. 


Fig. 4 Example of a Safety Belief Structure 


= 


ee 


Openness. This belief structure represents the perceived abil- 
ity to understand the potential trustee. In human interpersonal 


trust, this includes a judgment of the trusted agent’s honesty, 
forthright communication, and other behavioral and character 
attributes. With respect to automation, openness is often syn- 
‘onymous with “transparency,” ic., the ability to understand 
what a device does and how it works, Madsen and Gregor 
[27] found that the perceived understandability and the per- 
ceived technical competence of a system were key principal 
components in trust of automation. Figure 5 illustrates a 
simplified example of an openness belief structure. 
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Fig. 5 Example of an Openness Bel 
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2.2 Modulation of the Trust Evaluation of Belief Suuctures 


The evaluation of the trustworthine: 


of an agent does not 


‘occur in an abstract realm. It occurs only when the Trustor 
anticipates a making a decision on whether to delegate or 
not to a potential Trustee. Such a dec 


mn exists relative to 
the Trustor’s goals and motivations, i.e., something to be ac- 
complished. Accomplishment of goals occurs in larger social 
and situational context that includes affordances, obstacles, 
other actors, and often there exists substantial uncertainty, 
For example the social context as well as the perceived role 
of actors affects human-machine trust [19, 46]. 

Situational factors, in combination with individual per- 
sonality attributes, play a significant role in how a Trustor 
will perceive and evaluate risk, and estimate the impact po- 
tential consequences of a Trustee’s failure to perform. As 
perceived by the Trustor, risk has several components that 
may vary in importance and salience as a function of the s 
uation (including the task at hand) and individual attributes, 
Multidisciplinary studies identified the following key compo- 
nents: performance risk, financial risk, social risk, physical 
risk and psychological risk [22] as well as the risk of lost op- 
portunity and time [41]. When the environment and specific 
circumstances are perceived as reliably “safe” the magnitude 
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of the perceived risks associated with delegation to a Trustee 
are reduced [14]. 

In any given situation, individual factors of the Trustor 
can greatly affect the salience, importance and evaluation 
of the belief structures related to trust. For example, behav- 
ioral research has found that intuitive and affective processes 
create systematic biases that profoundly affect human trust, 
behavior, and choice [13, 40, 43, 44, 48]. Individual person- 
ality traits, as well as affective state, can affect delegation 
to autonomous agents [10, 11, 44]. Cramer et al [10] in a 
study of vehicle assistive information systems showed that 
human trust in the technology as reported by participants was 
strongly related to personality type, the situational context, 
and the manner in which the autonomous agent presented 
information. Oleson et al [36] provided further support that 
the principal influences on human trust of a robot partner 
included attributes of both the human and the robot as well as 
the environment (situation) in which cooperative work was 
to be performed. 


3 Method 


This study used survey research methods for investigating 
the belief structures related to a choice to rely upon an au- 
tonomous agent. The design of the survey instrument is dis- 
cussed below in detail. The participants were drawn from 
the population of individuals involved in any stage of life- 
cycle of intelligent, autonomous agents. We targeted this 
group of people in particular since their role is pivotal to 
the adoption of autonomy technology. Our survey was ad- 
ministered online via a commercial service. Survey data was 
downloaded, anonymized, verified, and otherwise system- 
atically pre-processed (as described below) in advance of 
anal 


3.1 Participants 


Our target population consisted of stakeholders and subject 
matter experts in autonomous systems. They could be in- 
volved in any stage of the lifecycle of an autonomous system, 
from design to test, to deployment decision, to operation or 
interaction. The technology of autonomy, while progressing 
rapidly, is still in its infancy. Our decision to target autonomy 
stakeholders and subject matter experts reflects the obvi- 
ous fact that they are driving development and deployment 
of autonomy technology. They will therefore heavily influ- 
ence early decisions regarding how and when to employ au- 
tonomous agents in real applications. Their attitudes regard- 
ing the trustworthiness of autonomous agents will profoundly 
shape attention and the perceived suitability of autonomous 
agents for many applications of significant interest. Under- 
standing these factors will contribute to initial autonomy 


designs that are likely to receive stakeholder trust. Therefore 
we prioritized consideration of this particular population over 
others. Our target population should not be confused with 
the end-users of autonomy technology whom may or may 
not attach equivalent importance to the various belief’s about 
trustworthiness. Understanding the unique trust-related fac- 
tors of end-users will be essential for long-term successful 
operations of intelligent, autonomous agents. We did not set 
out nor expect to obtain results that would necessarily apply 
to this larger group. 

Participants were recruited by personal solicitation in pro- 
fessional settings and in online forums related to autonomy 
technology and applications. Our plan for statistical analysis 
using conservative measures with no assumptions regarding 
distribution dictated a minimum sample size of twenty-nine 
participants. Of the sampled thirty-eight participants, we re- 
ceived valid survey responses from thirty-one. A valid survey 
response was one where every mandatory question was an- 
swered. Responses with missing answers or instances where 
participants ended their participation before completing the 
survey were omitted from analysis. Participant responses con- 
firmed that we did in fact sample from our target population. 
The sample population was a tech-savvy group that scored 
uniformly high on acceptance of innovation (see “Individual 
Innovativeness (1)” below and the results section for further 
discussion). Table 1 summarizes the basic demographics of 
our sample. Ultimately, we determined in analysis that the de- 
mographics of our sample were not a significant factor other 
than to indicate we had successful sampled a representative 
group from our target population 


3.2 Survey Design 


The survey was designed to elicit attitudes, opinions, and pref- 
erences that should shed light on belief structures of potential 
importance for a decision to rely on autonomous agents (see 
the earlier discussion of trust-related belief structures). The 
design of the survey was further guided by requirements to 
collect both context-free and context-dependent assessments 
from participants regarding the importance of twenty-eight 
specific hypothetical qualities of autonomous agents (see 
Table 2). For example, a specific quality might be: “The au- 
tonomous agent's behavior conforms to expectations.” This 
quality is a component belief of the belief structure called Pre 
dictability. The belief structures examined in the survey are 
those discussed earlier: Competence, Predictability, Safety, 
and Openness. To avoid introducing any particular bias it 
might evoke, we did not use the word “trust” anywhere in the 
survey. However, by targeting the belief structures and their 
‘components that have previously shown by other studies to be 
antecedent to trust we can be confident that the specific sur- 
vey questions are related to trust even if they do not mention 
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it explicitly. Furthermore, the broad notion of trust, an unob- 
servable mental state and psychological phenomena, is less 
interesting to us and other developers of autonomy technol- 
ogy than are the beliefs that ultimately result in a disposition, 
intention, and behavior to delegate a task to an intelligent 
autonomous agent. Therefore, these beliefs (and structures of 
beliefs) are of more concern because their specificity may be 
used to address questions of human delegation and reliance 
directly in terms of the form, function and operation of an 
autonomous agent. 


Fig. 6 From Original Trust Studies to Survey Question Design 


Meta-Analysis 


Muttiiscpinry Studies 


‘tudes of Trust, 


felis Itereatonship 


Test Beli 
Categories 


I 


Belief Structures 


{ Survey Questions } 


In addition to collecting demographic information typical 
of these types of surveys, we sought to gather data on indi- 
vidual factors that might influence reliance on autonomous 
agents, including personality traits, attitude toward innova- 
tion, and acceptance of risk. These were assessed using stan- 
dard personality inventories, as described below. 

Context-free participant attitudes toward the importance 
of autonomous agent qualities were assessed using a series of 
twenty-eight Likert-scale questions, one for each characteris- 
tic of interest as shown in Table 2. Context-sensitive attitudes 
were assessed with six systematically different scenarios. Sce- 
narios varied in terms of type and magnitude of risk as well as 
conflict and concord with the four belief structures described 
earlier. The scenarios challenged participants to choose: (a) 
to rely upon an autonomous agent, (b) another human, or (c) 
“either” to accomplish a given task of importance. The sce- 
narios are briefly described below and are discussed at length 
in [2]. In conjunction with their choice in each scenario, the 
survey queried participants using a Likert-scale regarding the 
relative importance of each of the four trust-related belief 
structures of interest (Competence, Predictability, Safety, and 
Openness). 

We measured “source credibility” of the agents using a 
standard instrument that assesses interpersonal trust-related 
attitudes [31]. The instrument is intended to what par- 
ticipants perceive as the “ethos” of another person, group 


Trust Beets 


‘Table 1 Demographic information for survey participants? 


Education 


Bachelors Degree 4 

Masters Degree 12 

PhD 12 

Age Employment 

<2 4 Education 10 

26-35 «7 Business 13 

36453 Government 1 

46-5510 Military 1 
56-65 6 
266 1 


* Totals are unequal since demographic questions were not mandatory. 


or organization, Participants were asked to assess all the au- 
tonomous agents presented in the scenarios collectively as a 
group. These questions provided us with three additional mea- 
sures of inter-correlated constructs: Trustworthiness, Compe- 
tence, and Caring/Goodwi 

The survey included a variety of question types, typical 
of social science research, e.g., dichotomous questions, rank 
orders, Likert scales, semantic differentials and opportunities 
for free-form narrative response. We now review the details of 
the survey design in the same order in which they appeared 
to participants in the actual survey. The complete survey, 
scenarios, and other related materials are available for review 
and use by other researchers on request to the authors or as a 
supplement to this paper from the publisher. 


3.2.1 Survey Part 1: Personality Inventories 


Participants completed three standard personality survey 
instruments used in the social sciences: Big Five Inven- 
tory (BF1), Individual Innovativeness (11), and the Domain- 
Specific Risk Taking Scale (DOSPERT). 


Big Five Inventory (BF1). This study utilized a 10 item short 
version of the Big Five Inventory (BFI-10) developed by 
Rammstedt and John [39]. The full Big Five Inventory (BFI- 
44) is a multi-dimensional personality inventory [5] that mod- 
els personality traits and defines five relatively distinct ar- 
as of individual differences: Openness to new experience, 
Conscientiousness, Extraversion, Agreeableness, and Neu- 
roticism [23]. 


Individual Innovativeness (II). The Individual Innovative- 
ness (II) scale [21] assesses an individual’s orientation to- 
ward “change,” where change is a result of an new idea, 
object, practice that is perceived as an innovation. This ori- 
entation predicts how likely the individual is to accept an 
innovative change early in the adoption process. A high score 
indicates an individual is an “early adopter.” 
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‘Table 2 ‘Twenty Eight Hypothetical Trust-Related Quali 


sof Intelli 


-nt, Autonomous Agents? 


Category Name (Quality Description 
Competence Capable ‘The autonomous agent can achieve a desired result 
Knowledge The autonomous agent has all the knowledge it needs to do its job 
Accurate What the autonomous agent believes to be true is actually true. 
Skilled ‘The autonomous agent possesses good methods for using its knowledge to do its task 
Logical The autonomous agent reasons correctly according to logic 
Heuristic When it cannot figure out something using logic, the autonomous agent can make good guesses. 
Corrective ‘The autonomous agent recognizes gaps in its knowledge and tries to learn what it needs to know. 
Adaptive The autonomous agent learns to correct its mistakes, as well as to improve and maximize its capability, 
Predictability Expected The autonomous agent's behavior conforms to expectations. 
Purposeful The autonomous agent purposefully acts to achieve goals. 
Helpful ‘The autonomous agent will assist people, whenever it is possible. 
Directable ‘The autonomous agent accepts and carries out orders 
‘The autonomous agent uses its knowledge and skills in expected ways. 
Safety ‘The autonomous agent's behavior will not harm humans or human inte 
Any incorrect behavior by the autonomous agent will not cause harm. 
‘The autonomous agent fails gracefully and recovers from its failure promptly. 
‘The autonomous agent adheres to obligations, principles, and rules. 
Correctable ‘The autonomous agent can correct its own defects or they can be corrected by a human. 
Protective ‘The autonomous agent recognizes and avoids harming humans’ interests. 
Favorable Given alternatives in what to do or how to do it, an autonomous agent will act in a way that is favorable to a 
human being who might be affected. 
Openness Visible What the autonomous agent is doing and how it works is easy to see and understand, 
Honest ‘The autonomous agent believes what it says. 
‘Transparent Its easy to inspect an autonomous agent. 
Communicative ‘The autonomous agent communicates in a way that is easy to understand, 
Interactive The autonomous agent responds when you are trying to communicate with it 
Attentive ‘The autonomous agent is aware of communication between others nearby. 
Reactive The autonomous agent responds quickly to calls for attention. 
Disclosing The autonomous agent communicates truthfully and fully. 


© Quality Description is shown exactly as it app 


wred in the survey. 


Domain-Specific Risk Taking Scale (DOSPERT). The DOS- 
PERT Scale [6] evaluates the likelihood that a participant 
might participate in risky activities or behaviors, and risk- 
perception, across five major domains: Ethical, Financial, 
Health/Safety, Social, and Recreational. These are the basis 
for sub-scale scoring. Both risk-taking (self-reported likeli- 
hood) and risk-perception (“gut i utilize a 7-point 
scale for ratings. Participant ratings are added across all the 
items of a given domain sub-scale to obtain scores. Higher 
scores suggest greater risk-taking in a domain, or percep- 
tion of greater risk, respectively. Internal consistency, con- 
struct validity, test-retest reliability, and other quality tests of 
DOSPERT were evaluated by Weber et al [47]. The 30-item 
model used in this study was developed and tested using 
confirmatory factor analyses [6]. 


3.2.2 Survey Part 2: Importance of Agent Qualities 


Participants were asked to rank the individual importance of 
twenty-eight different statements about the qualities of intel- 
ligent, autonomous agents; all are arguably anthropomorphic 
characteristics. These qualities represented a composite of 
tust-related agent characteristics derived from published 
studies as well as additional hypothetical qualities devised 


by us. Each of these qualities can be considered a member of 
one of four trust-related categories derived from our literature 
review: Competence, Predictability, Safety, and Openness. 
However, we could not ask participants directly about “trust” 
without risking introduction of bias of interpretation of the 
word. Our strategy was to ask participants to rate each qual- 
ity with respect to its importance for a “good” autonomous 
agent. Conceptually, “goodness” encompasses all of the cat- 
egories of qualities related to trust that were the target of 
our survey. The ranking of importance for each quality was 
assessed using a Likert scale: “Not at all important,” “Slightly 
Important,” “Somewhat Important,” “Important,” and “Very 
Important.” Participants were also given the option of “Can- 
not Decide,” in which case their response was omitted from 
the corresponding analyses. At this point participants were 
not aware of the scenarios to be presented later in the survey 
and so their answers reflect their attitudes ab initio. The full 
list of trust-related qualities, organized by category, is given 
in Table 2. 


3.2.3 Survey Part 3: Reliance Challenge Scenarios 


The third part of the survey was organized around six chal- 
lenge scenarios in four application domains: transportation, 
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finance, healthcare, and disaster management [2]. Although 
on initial reading these scenarios might appear to some peo- 
ple to be fanciful, futuristic and unlikely to be encountered by 
an “average person” in “real life,” they in fact are representa- 
tive of actual current or proposed applications of intelligent, 
autonomous systems. The pool of participants expressed no 
incredulity regarding the scenarios in their free narrative 
response opportunities. 

Each scenario was designed to systematically vary and 
cause conflict between the hypothetical belief structures of 
interest. Additionally, the scenarios addressed different types 
of risks with potential negative consequences ranging from 
low to medium or high impact. Our intention in the use of 
scenarios was exploratory, not to test any specific hypothe- 
sis. Therefore, given the length of the survey, we deemed it 
too much of a demand on participants to exhaustively rep- 
resent all combinations of the potentially relevant variables 
in individual scenarios. We considered that the criteria for 
successful use of the challenge scenarios to be a finding of 
significant difference within or between participants across 
all the scenarios. Such a finding would fulfill the exploratory 
purpose of identifying particular phenomena for follow-on 
study. 

In each scenario, participants faced a dilemma in a forced 
choice of whether to delegate to an autonomous system, to a 
human, or to “either” (ie. no preference, which we scored as 
an allowable choice for an autonomous agent). Following the 
participant's choice in each scenario, they were asked to rank 
the relative importance of the four trust-related categories 
to their decision. Each of the scenarios is briefly described 
below. Only partial text of the actual scenario is included 
here for the sake of brevity.! 


Airport Transportation (Robo-Taxi). This scenario was de- 
signed to be a choice with relatively low-risk consequences 
(physical, performance, time-loss). The agent is elaborated 
as being of medium competence and medium predictability 
according to reputation. Further, it is described as easy to 
observe and explain what it is doing. 


“You have just flown into the airport of a large, unfa- 
miliar city whose streets are teeming with cars and 
people. It is rush hour, and needing transportation to 
your hotel, you walk to the taxi stand only to discover 
that you have a choice of a human-driven taxi or a 
driverless Robo-Taxi...” 


Financial Management (Robo-Trader). This scenario was 
designed to be a choice with primarily social and financial 
risks at a low level. The competence and predictability of the 
agent are relatively high, based on reputation. Itis difficult to 


' The full scenario descriptions are available upon request to the 
corresponding author or as a supplemental download. 


observe how the Robo-Trader actually reaches decisions but 
it is willing to try to explain, 


“You have been appointed trustee of a family mem- 
ber’s estate. Your duties include choosing how to 
wis . Your personal money 
is not at risk. However, a poor investment decision 
could cause the trust to lose money and will strain 
your family relations. You can choose a stock bro- 
ker who personally selects and trades all stocks in 
the trust’s portfolio. Alternatively, you can choose a 
stock broker who relies heavily on a Robo-Trader.” 


Medical Procedure (Robo-Surgeon). This scenario presents 
a choice between a highly competent and predictable Robo- 
Surgeon, an “expert” in the surgery required, versus a com- 
petent but non-expert human physician. The primary risks 
are physical and performance and they are high. 


“You have just suffered a major sports-related injury. 
You have torn the bicep tendon in your shoulder. If 
the damage is not repaired quickly and correctly, you 
will permanently lose mobility and strength in the 
arm..." 


Home Healthcare (Robo-Caregiver). This scenario repre- 
sents a moderate financial and social risk. The robotic agent 
has moderate competence but its predictability is unknown, 
However, it is easy to observe and explain what it is doing. 


“Your elderly mother has been diagnosed with a de 
generative medical condition and you are responsible 
for making medical decisions on her behalf. Your 
mother needs assisted living with someone in your 
mother’s home at all times. ...” 


Disaster Response (Auto-FirstResponder). This scenario 
poses high physical and social risks by a moderately pre- 
dictable and competent agent. Owing to situational factors, 
it will be difficult to observe the agent’s actions. Sending 
in a human team may result in additional casualties given 
situational uncertainty. 


“A major disaster has just occurred and you are the of- 
ficial in charge of responding. A train has derailed in 
a populated suburban neighborhood and there are 
reports that the train was carrying hazardous bio 
chemical materials. ...” 


Lost At Sea (Emergency-AutoCaptain). This scenario is a 
high physical and performance risk where there is uncertainty 
about the competence and predictability of the autonomous 
agent due to damage it may have sustained. Otherwise, it 
is known to be at least moderately competent and readily 
communicative about what it is doing and why. 
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“You have just been involved in a terrible boating 
disaster while sailing deep in the South Pacific. The 
captain, the crew, and most of the passengers are 
either dead or lost at sea. ...” 


3.2.4 Survey Part 4: Agent “Source Credibility” 


As a third check of participant attitudes, we used another 
standard instrument designed to measure “source credibility” 
[31]. Typically, this instrument is used in teamwork envi- 
ronments to assess interpersonal trust-related attitudes. It 
assesses what participants perceive as the “ethos” of the per- 
son in question, (e.g., the boss). In our survey, participants 
were asked to assess the autonomous agents presented in 
the scenarios, considered as a group. These questions pro- 
vided us with three measures representing inter-correlated 
constructs (as defined by the instrument): Trustworthiness, 
Competence, and Caring/Goodwill. 


3.2.5 Survey Part 5: Demographic Information 


The final portion of the survey included demographic ques- 
tions to help us evaluate how well we achieved our targeted 
population. Questions were non-mandatory and addressed 
the following subjects: gender, age, highest academic degree, 
relevant experience, type of employer, and job title. 


3.3 Procedure 


The survey was administered online via a commercial ser- 
vice.? With certain restrictions, this service offered the nec- 
essary and sufficient tools for constructing the survey, admin- 
istering its application, and for data collection. The technical 
limitations on survey design imposed by this service are 
discussed in the Limitations section below. The survey was 
conducted over the period of eight weeks. Collected data 
was retrieved from the online service in a suitable file format 
and put under configuration control to enable potential re- 
analysis later. Prior to analysis, the data was pre-processed 
to put it in a suitable format. Invalid records, such as those 
resulting from a participant not completing the survey or 
skipping mandatory questions, were identified, labeled, and 
excluded from this analysis. Analysis of indecision leading 
to skipped questions is a topic for follow-up. Finally, within- 
test scoring for each of the standard personality instruments 
was calculated and added to each valid record. All person- 
ally identifiable information was stripped and code numbers 
substituted to ensure participant anonymity. 


2 nttp://polldaddy.com 


‘Table 3. Top Three Most Important Autonomous Agent Qualities Re- 
ported by Participants 


Rank 


me Quality Description 


Safe ‘The autonomous agent’s behavior will not harm hu- 
mans or human interests 

‘The autonomous agent can achieve a desired result, 
Any incorrect behavior by the autonomous agent will 


not 


2"! Capable 
3! Limited 
use harm, 


Analysis. Analysis of the data included standard statistical 
descriptive measures. The two-tailed Pearson Product Mo- 
ment Correlation (PPMC) r was chosen as a conservative test 
of the relationship among the variables in the survey. The 
PPMC r was computed individually between every agent 
quality, the categories of qualities, demographics, and the 
participants’ choice of agent and related answers by scenario. 
For all results reported here, the correlations r are significant 
with 95% confidence: o < 0.05, N = 31, df = 29. Critical 
values for significance were obtained by table lookup. Results 
with a confidence of 98% and 99% are indicated by * and 
** respectively. Rankings of the relative importance of agent 
qualities were computed by analysis of answer frequency 
distributions. Additional descriptive statistics were computed 
within-participants and for the group. 


4 Results 


The results of the survey are presented here in the context of 
the specific questions we sought to answer: 


— Which anthropomorphic beliefs about the qualities of an 
autonomous agent do participants report as most impor 
tant to a decision to rely upon one? 

= Do their actual choices, given a particular scenario, corre- 
late with beliefs they self-report to be important or with 
other beliefs reported as less important? 

= What is the relative importance among such beliefs (both 
self-report and actual)? 

= Do those beliefs and/or their importance vary by indi 
ual personality or situational factors? 


Most Important Beliefs. Recorded prior to participant 
posure to the specific use case scenarios, the top three agent 
qualities reported as being required for a “good” autonomous 
agent are consistent with what was expected based on the 
social science literature for trust. These are shown in Table 3. 
Surprisingly, the top three qualities were not significantly 
correlated with any of the actual delegation choices made in 
any scenario (see Table 4). In other words, those qualities 
of “good” intelligent, autonomous agents reported by partici 
pants to be the most important had statistically insignificant 
impact when it came time to actually make a decision to 
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‘Table 4 Importance of Qualities of Autonomous Agent Significantly Correlated with Actual Participant Reliance on Autonomous Agent® f 


‘Airport Trans. Medical Proc. Home Health, Disaster Resp. Lost at Sea 
Corrective, r= 0.396 Accurate, r = —0.405 none Visible, =0.437* — Corrective, r= 0.418% Protective, = 0.419* 
Heuristic, r = 0.395 Visible, r = —0.390 
Attentive, r=0.393 Disclosing, r= 0.375 


© Pearson Product Moment Correlation, a < 0.05, N = 31. dl 
See Table 2 for actual quality descriptions used with participants 


become reliant on an autonomous agent in a specific, hypo- 
thetical scenario. 


Trust-Related Quality Categories. In each scenario, partici- 
pants were asked to rank the importance of the quality cate- 
gories Competence, Predictability, Safety, and Openness to 
their choice of whether to rely on the human, autonomous 
agent, or either. As shown in Table 5, the importance of three 
of the twenty-eight specific qualities were significantly cor- 
related with one or more of these four categories in three of 
the six scenarios. That no one category correlated with all 
scenarios suggests the importance of situational factors in 
evoking the salience of particular agent qualities to a choice 
regarding reliance. 


Risk and Benefit. After participants indicated their choice 
of whether to rely or not on an autonomous agent in each 
scenario, they were asked to assess the magnitude of risk 
and benefit (type or source were left unspecified). These 
sessments were found to be correlated (positively or nega- 
tively) in five of the six scenarios as shown in Table 6. As a 
sample which proved to consist of “innovators” and “early 
adopters” (to be discussed below), it is perhaps unsurprising 
that perceived benefit and risk played an important role ina 
participants decision regarding reliance. 


Source Credibility. Following their consideration of the six 
scenarios, participants also answered questions about their 
attitudes towards the autonomous agents considered as a 
group. As a reminder, for this purpose the survey employed 
the Source Credibility instrument, developed by MeCroskey 
and Teven [31], that provides ratings for factors of Trust- 
worthiness, Competence, and Caring/Goodwill (not to be 
confused with the Competence category of agent qualities). 
Taken overall, in the instrument the three factors are intended 
to represent the participants’ perception of the autonomous 
agent’s “ethos.” 

Although the source credibility results were not corre- 
lated with the reliance choices in any scenario, all three fac- 
tors correlated at the 99% confidence level with the two 
specific agent qualities as shown in Table 7. The Competence 
factor is interpreted to reflect the participant's perception 
of the agent’s knowledge and ability to use that knowledge 
in specific, relevant domains. The Caring/Goodwill factor 


}: * indicates e < 0.02. 


is interpreted in the context of “intent toward the receiver” 
based on a perception of understanding, empathy, and re- 
sponsiveness. The Trustworthiness factor is interpreted as 
a perception of honesty and related traits. Previous studies 
have shown these last two factors are highly predictive of 
“believability” and “likeableness.” We interpret these results 
as indicating the importance of the contribution of the two 
agent qualities to an overall positive perception of the ethos 
of the hypothetical autonomous agents presented in the study 
scenarios. These results are shown in Table 7. 


Personality Factors. We anticipated a systematic variation 
between relative preferences for competence and predict- 
ability correlated with personality measures, e.g., risk toler- 
ance, openness to innovation, and participants’ perception of 
risks in each scenario, Indeed, the most significant correla- 
tions of agent quality importance with choice of human or 
autonomous agent varied both by scenario and by participant 
personality factors. These results are shown in Table 8. In- 
dividual personality factors appear to influence the choice 
to become reliant on an intelligent, autonomous agent de- 
pending on the details of specific use case scenarios. These 
personality factors include likelihood of accepting innova- 
tion, perception and acceptance of different types of risks, 
and factors such as Extraversion, Openness, and Conscien- 
tiousness. 

The negative correlation of Innovation II with selection 
of an autonomous “Robo-Trader” in the Financial Manage- 
ment scenario might be understood in the context of recent 
real-world events. In the past few years, there have been sev- 
eral instances where automated trading systems have caused 
massive market losses for the financial firms running those 
systems. The sample population was a tech-savvy group that 
scored uniformly high on acceptance of innovation.* Thus, 
they are likely to understand the present technical limitations 
in regards to predictability of autonomous agents and conse- 
quent higher risk present in the Financial Management and 
Lost At Sea scenaris 


Relative Importance of Trust-Related Quality Categories. 
As discussed earlier, four broad categories of belief about 
other agents that are important to trustworthiness emerge 


3 ‘The Il factor group mean score was 74.5, wil 
strong mix of “innovators” and “early adopters.” 


variance indicating 
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‘Table 5 Qualities of Autonomous Agent Significantly Correlated with 
ri 


Trust Categories by Scenario’ 


‘Table 7 Qualities of Autonomous Agent Significantly Correlated with 
Agent Source Credibility Factors! 


Name? vs. Category _ Home Health. Dis 


ter Resp. Lost at Sea 


0.400 
0.449 


‘ible vs. Competence 
ible vs. Predictability 
‘ible vs. Openness 
Corrective vs. Safety r 
Disclosing vs. Openness 


Pearson Product Moment Correlation, at < 0.05, N= 
jee Table 2 for actual quality descriptions used with participants. 


Table 6 Correlation of Perceived Risk and Ben 
Autonomous Agent by Scenatio® 


with Choice of 


Scenario Risk Benefit, 

‘Airport Trans. r= —0.546** d 

Financial Man, NS NS 
—0.380 gst 
—0.470* 632¢* 


4gaee 
ssse* 


Disaster Resp. 
Lost at Sea 


© Pearson Product Moment Correlation, o < 0.05, N 
** indicates a < 0.01 
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from previous studies: Competence, Predictability, Safety, 
and Openness. In each scenario following their choice of 
whom to rely upon (human, autonomous agent, or either), 
participants were asked to rate the relative importance of 
these four categories to their decision using a Likert scale. 
Participants were encouraged to make an absolute ordering, 
but ties were allowed. Table 9 shows the rank order for each 
scenario computed as a mean score across participants 

As noted earlier (See Table 5 and accompanying discus- 
sion), the self-reported relative importance of the trust-related 
quality categories had a statistically significant correlation 
with participants’ choice of agent to rely upon in three of the 
six scenarios, and these varied by scenario. A comparison 
of the mean scores and standard deviations across partici- 
pants within each scenario, and across scenarios, yields these 
additional observations: 


= The relative cross-scenario ranking of Competence and 
Safety are indistinguishable, with the exception of the 
Home Healthcare scenario where Safety is one standard 
deviation above Competence. Overall, this pair is ranked 
as most important to participants’ choice of agent in all 
scenarios. 

— The relative cross-scenario ranking of Predictability and 
Openness are also indistinguishable with the sole excep- 
tion of the Lost At Sea scenario where Predictability is 
one standard deviation above Openness. 

— Small differences in ranking of agent quality categories 
are evident across scenarios and this may be worthy of 
further study. 


Name? Trustworthiness Competence Caring/Goodwill 
Heuristic r= 0,506"* 
Corrective r= 0.466** 


* Pearson Product Moment Correlation, a < 0.05, N= 31, df 
* indicates a < 0.01 
‘Table 2 for actual quality descriptions used with participants. 
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‘Table 8 Participant Personality Factors Significantly Correlated with 
Reliance on Autonomous Agent® 


Scenario Correlated Personality Factor(s) 
‘Airport Trans, none 
Financial Man. Innovation I 0.355 
Medical Proc. BFI Extraversion, 0.368 

BFI Openness, = 0.366 
Home Health. DOSPERT Social Risk. 0.364 
Disaster Resp. BFI Conscientiousness,r = 0.366 
Lost at Sea Innovation II, r = —0.366 


® Pearson Product Moment Correlation, « < 0.05, N 
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5 General Discussion 


When considering the twenty-eight specific qualities of in- 
telligent, autonomous agents related to trust (see Table 2), 
the study found that the top three agent qualities cited by 
participants as the most important for delegation and reliance 
upon an autonomous agent (see Table 3) are consistent with 
two general qualities highlighted by previous interpersonal 
trust and human factors studies, i.e., (1) the ability of the 
machine to achieve the desired results, and (2) not causing 
harm. 

Surprisingly, those top three qualities, consciously se- 
lected, were not significantly correlated with any of the ac- 
tual delegation choices made by participants later in the sur- 
vey when they considered specific use-case scenarios. Other 
lesser important qualities proved to be better predictors, and 
these varied by scenario (see Table 4). Our interpretation 
is that there may exist an influential disposition of beliefs 
regarding trustworthiness that are not necessarily the most 
salient during conscious introspection. Secondly, the context- 
independent responses differ from those following presen- 
tation of a challenge scenario because the participants are 
forced to examine their trust-related beliefs in a specific sit- 
uation. These differences could prove important for future 
requirements and evaluation of autonomy technology; both 
context-independent and specific belief’ are likely to be im- 
portant. 

We expected Competence and Predictability to be re- 
ported by participants as the most important quality cate- 
gories for trust. This was confirmed. However, only one or 
two of these four categories proved in any single use-case 
scenario to be a good predictor of choice of reliance on an 


12 


David J. Atkinson, Micah H. Clark 


‘Table 9 Participant Report of Rank Order by Importance of Autonomous Agent Quality to Participants De: 


on to Become Reliant on an Agent 


Scenario Category Rank Order Grand Mean SD 
‘Airport Trans. Safety, = 3.69 Competence, t= 3.56 Predictabilin Openness,3=238 3.05 0.67 
Financial Man. Competence, ¥ = 3.56 Safety, % = 3.06 Openness, % Predictability, 293 073 
Medical Proc Safety, =3.75 Competence, ¥=3.12  Predictabilin Openness, 3.05 0.79 
Home Health. Safety, =3.78 Competence, ® Predictability Openness, % = 3.20 0.45 
Disaster Resp. Competence, ¥ = 3.56 Safety, Predictability Openness, % 3.03 0.53 
Lost at Sea Competence, % = 3.66 Safety, Predictability Openness, ¥=2.16 3.00 0.69 


® Rank proportional to Mean Score ¥ across part 


ipants (4 


intelligent, autonomous agent, and then in only in three of 
the six scenarios (see Table 5). 

Two of the most predictive qualities were also strongly 
correlated with participants’ overall assessment of the “ethos” 
of the intelligent, autonomous agents in the scenarios (per 
measured Source Credibility factors; see Table7). This pro- 
vides further evidence for the importance of these qualities 
to an overall positive perception of the agent. 


We conclude that self-report of the individual importance 
of specific trust-related qualities of intelligent, autonomous 
agents without context are poor predictors of a decision to 
rely upon such an agent when a person is confronted with 
a choice in a specific hypothetical use-case scenario. Other 
qualities appear to be elevated in importance in specific sce- 
narios based on a combination of situational and personality 
factors. This points towards a need for future studies that 
examine trust-challenging use case scenarios using meth- 
ods that achieve greater realism. In particular, such scenarios 
should potentiate affective responses that cannot be examined 
easily using surveys or unrealistic in-laboratory experiments. 
Specific individual personality factors were shown to influ- 
ence the choice to become reliant on an autonomous agent. 
These include likelihood of accepting innovation, perception 
and acceptance of different types of risks, and factors such 
as Extraversion, Openness, and Conscientiousness. 

The importance of these results for developers of intel- 
ligent, autonomous agents, pethaps embodied as robots, is 
likely to have a significant impact on human-robot interaction 
design choices. Understanding how the population of inno- 
vators and early adopters targeted by this study respond to 
challenging questions of trust will be critical for acceptance 
and deployment in critical applications of broad interest such 
as rescue robotics. Further investigations are required to un- 
derstand the degree to which these results extend to different 
populations who might reasonably be expected to interact 
with intelligent, autonomous agents, in particular naive users. 


Technology developers should consider how the quali- 
ties of Competence, Predictability, Safety and Openness of 
intelligent, autonomous agents are accurately measured and 
portrayed in the human-robot interface. Portrayal must be 
done in such a manner that it correctly evokes human interper- 
sonal trust evaluative processes and furthermore, contributes 


Somewhat Important; 1 = Not at all Important 


to well-calibrated trust and appropriate reliance in specific 
challenging application scenarios. Related follow-on research 
by the authors is underway using an immersive simulation of 
a disaster environment and interaction with a social robot to 
improve the methodology and further examine attributions 
of trustworthiness [3]. 


Key Points: 


= We did not confirm any scenario-independent specific 
agent qualities that uniformly contributed to an affirma- 
tive human reliance decision. Certain qualities were im- 
portant in some scenarios and not in others. 

Specific qualities of agents, and categories of those quali- 
ties, are likely to be raised or lowered in importance de- 
pending on both situational (application-specific) factors 
and human psychological factors. Further investigation 
is required to identify those qualities and provide a mech- 
anism for understanding how their role and importance 
in human reliance decisions changes. 

Certain personality factors, including high scores for Ex- 
traversion, Openness, and Conscientiousness are very im- 
portant in some situations while in others, the tolerance 
for certain kinds of risk is dominant when it comes to 
deciding to become reliant on an intelligent, autonomous 
agent. This suggests that certain people may more readily 
accept a dependency on an agent, and conversely, others 
are likely to be extremely resistant, Heretofore, most re- 
search on human-robot interaction has not considered the 
importance of individual differences. 


6 Limitations 


We acknowledge several limitations in our study that taken 
individually and as a group should provide caution regarding 
extrapolation of these results to other or larger populations. 
Despite these limitations, however, the exploratory nature of 
our study provides useful data and succeeds in provoking a 
need for further research, 


Sample Population. Our sample of subject matter experts 
may not be representative of the larger population and this 
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adds uncertainty to our conclusions. The familiarity of this 
group with the technology of intelligent, autonomous agents 
may bias their attitudes, inoculate them from some degree of 
unconscious anthropomorphism, and increase their overall 
distrust. This question deserves investigation. 


Agent Qualities. The twenty-eight qualities of intelligent, 
autonomous agents are described using words that may have 
different meanings to different sample populations. As such, 
they can only be approximations of the semantics of actual 
beliefs about agents, 


Statistical Measures and Error, While we have applied ap- 
propriate statistical analysis to the data in our study and been 
as rigorous as possible with control of the data, it is possible 
that errors have been introduced from various sources. These 
include participant error, programming error and data coding 
error. 


Survey Technical Limitations. ‘The automation used for de- 
velopment and administration of the survey did not permit 
certain procedures that are often used to improve validity of 
the data collected. Specifically, the survey questions were in 
fixed order of presentation, identical for all subjects and not 
randomized, Secondly, the survey questions were not able to 
use a randomized mixture of positive and negative phrasing. 
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Abstract 


‘The central thesis of this paper is that the technology of 
intelligent, autonomous machines gives rise to novel fault 
modes that are not seen in other types of automation. As a 
consequence, autonomous systems provide new vectors for 
cyber-attack with the potential consequence of subversion, 
degraded behavior or outright failure of the autonomous 
system, While we can only pursue the analogy so far, 
maladaptive behavior and the other symptoms of these fault 
modes in some cases may resemble those found in humans. 
‘The term “psychopathology” is applied to fault modes of the 
human mind, but as yet we have no equivalent area of study 
for intelligent, autonomous machines. This area requires 
further study in order to document and explain the 
symptoms of unique faults in intelligent systems, whether 
they occur in nominal conditions or as a result of an outside, 
purposeful attack. By analyzing algorithms, architectures 
and what can go wrong with autonomous machines, we may 
a) gain insight into mechanisms of intelligence; b) lear 
how to design out, work around or otherwise mitigate these 
new failure modes; c) identify potential new cyber-security 
risks; d) increase the trustworthiness of machine 
intelligence, Vigilance and attention management 
mechanisms are identified as specific areas of risk, 


Introduction 


Psychopathology is the study of mental illness, mental 
distress, and abnormal or maladaptive behavior. It is the 
study of fault modes of the human mind. As yet, we have 
no equivalent area of study for intelligent, autonomous 
machines. Software engineering techniques for reliable 
systems are applicable (as they are to all complex software 
artifacts), but insufficient. The topic of this paper is the 
proposition that the technology of intelligent, autonomous 
systems gives rise to novel fault modes that are not seen in 
other types of automation, These fault modes arise from 
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the nature of the algorithms and how they perform in real- 
world situations (including human interaction) with 
uncertain data, As a consequence, autonomous systems 
may provide new vectors for eyber-attack that could lead 
to subversion, degraded behavior or outright system 
failure. 

This paper arose from a bit of fun the author was having 
by examining examples of “robots run amok” in popular 
literature and media, HAL 9000, of the movie “2001: A 
Space Odyssey” is a canonical example. These cases are 
often described in anthropomorphic terms related to human 
psychopathology, and this became the genesis of the idea 
for a psychopathology of intelligent machines. Although 
the analogy will stretch only so far, the search for 
intelligent machine near-equivalents of certain human 
mental disorders has already yielded a few insights that are 
described herein. The over-riding question is whether 
something like the behavior of these fictional malevolent 
machines could actually occur. In many cases, the answer 
is “probably not” but in a few, the answer is “probably 
yes.” If so, can we identify plausible mechanisms that 
explain the nature of the amok machines’ failures, given 
present artificial intelligence technology and what we can 
reasonably project on the horizon? 

This possibility suggests that there are fault modes for 
autonomous systems that remain unexplored and their 
implications unknown, The purpose of this paper is to raise 
that question explicitly, and to do so in the context of fault 
modes as potential vulnerabilities to attack, exploitation 
and subversion. 

We stand to gain certain benefits by analyzing the 
unique fault modes of autonomous systems. Such studies 
might provide insight into aspects of machine intelligence 
just as studies of human mental disorders have historically 
provided insight into the functioning of the brain, With the 
human mind, psychologists seek explanations for mental 
disorders from biological sources (relatively rare), innate 
biases, and faulty inference. Such failures of the human 


mind are often based in experience and learned behavior, 
including interpersonal communication and relationships 
with social and group effects. These are well documented. 
In contrast, with autonomous systems we must seek 
explanations for anomalous, maladaptive behavior in 
hardware (probably rare), software algorithms, logic, 
knowledge and situational uncertainty. Also guided by the 
study of human mental disorders, we should look for 
sources of machine intelligence fault modes in experience 
(episodic memory) and machine learning, including 
human-machine interaction and other aspects of social and 
affective computing 

Some of these autonomous system faults may occur in 
the course of day-to-day nominal operations and be easily 
“cured.” Of greater concern, it is possible that some 
psychopathologies of machine intelligence could be 
induced in a new form of cyber-attack, thereby creating 
new risks with potentially very serious consequences. We 
have the opportunity, now, to focus research on how to 
design out, work around or otherwise mitigate the failure 
modes we discover. It is best if this is accomplished sooner 
rather than later due to the potential adverse consequences. 
Ultimately, the real payoff for AI research and 
development of autonomy applications is the opportunity 
to increase the trustworthiness of machine intelligence. 
Today, this is cited as a chief obstacle to greater 
deployment of autonomous systems (Dahm 2010). 

The sections below provide essential background and an 
initial analysis of the symptoms and sources of selected 
example fault modes of autonomous, intelligent systems. 
In each case, we examine these fault modes with respect to 
vulnerability to cyber-attack, In the conclusion section, we 
discuss directions for future research and parameters of the 
required studies. 


Essential Background 


The technology of autonomous systems extends beyond 
conventional automation and solves application problems 
using materially different algorithms and software system 
architectures. This technology is a result of 
multidisciplinary research primarily in the fields of 
artificial intelligence and robotics, but drawing on many 
other disciplines as well, including psychology, biology, 
mathematics and others. Research on autonomous systems 
spans multiple areas, including (but not limited to) 
algorithms, computing theory, computing hardware and 
software, system architectures, sensing and perception, 
learning, and the acquisition and use of large stores of 
highly interconnected and structured, heterogeneous 
information 

The key benefit realized from autonomy technology is 
the ability of an autonomous system to explore the 


possibilities for action and decide “what to do next” with 
little or no human involvement, and to do so in 
unstructured situations which may possess significant 
uncertainty. This process is, in practice, indeterminate in 
that we cannot foresee all possible relevant information 
(ic., features and their relationship to one another) that 
could be a factor in pattern-directed decision-making, 

The autonomous ability to decide on next-steps is the 
core of what enables many valuable applications. “What to 
do next” may include a wide variety of actions, such as: a 
step in problem solving, a change in attention, the creation 
or pursuit of a goal, and many other activities both internal 
to the operation of the system as well as actions in the real 
world (especially in the case of embedded or cyber 
physical systems). Il-informed efforts to “envelope” or 
otherwise externally constrain the behavior of autonomous 
systems are sacrificing the most important strength of the 
technology — to perform in ways we cannot a priori 
anticipate. 

However, while the technology delivers new capabilities 
to perform work in a wide variety of under-specified and 
dynamic situations, it is also extremely complex to the 
point where conventional software systems test and 
evaluation methods are no longer sufficient to establish nor 
maintain confidence in autonomous systems. It is system 
complexity, arising from specific component technologies 
of autonomy (individually and collectively), that creates 
the prospect of new eyber-security risks. 

OF special importance is computational complexity: a 
measure of the resources required by a given algorithm to 
reach a result, Computational complexity is measured in 
time (e.g., wall clock time) and space (e.g., memory 
storage), and there are multiple other important attributes 
as well. The decision by an autonomous system of “what to 
do next” is the result of an algorithm that can be viewed, 
abstractly, as maximizing a utility function. These 
algorithms, intrinsic to autonomous systems, are typically 
of very high computational complexity; that is, they may 
require exponential amounts of time and/or space. 

Strict utility-based decision-making processes are 
recognized to be impossible in non-trivial domains (for 
people as well as machines). This is a result of the 
potentially infinite courses of action available, and the 
consequent inability to exhaustively analyze all of the near 
infinite number of possible system states; the inability to 
obtain and store all potentially relevant facts; and the 
intrinsically uncertain relationship between chosen actions 
and consequences when the environment has complex 
dynamics including other actors (Brundage 2014), 

Consequently, as a rule, the process of decision-making 
by an autonomous system is intrinsically limited by the 
available information, computational resources, and the 
finite amount of time available to reach a conclusion. This 
is referred to as “bounded rationality” (Simon 1958) and 


serves as a bedrock principle for research in artificial 
intelligence (Al), The result is that we can only hope to 
approximate optimal decision-making and behavior in an 
intelligent, autonomous system: “Satisficing” is acting in a 
way that leads to satisfactory and sufficient (“good 
enough”) outcomes, 

We conjecture that this heuristic, algorithmic struggle 
for computational resources with limited time and 
information is a principal source of novel fault modes that 
arise in autonomous, intelligent systems. 


Fault Modes 


What could possibly go wrong? That is the question asked 
by every researcher, developer, decision-maker, and user 
of an intelligent system, There exists the familiar panoply 
of software and system faults shared by all complex 
computational systems. Those are not our focus here, Our 
interest is in what new types of faults might exist by virtue 
of the nature of the algorithms in intelligent systems, or 
their application in certain circumstances, or as a result of 
malicious manipulation. Do such fault modes exist? 

The purpose of this section is to stimulate thought, 
discussion, and ideally, to convince you that the answer is 
likely to be “yes.” The existence of these fault modes 
arises directly from the limitations imposed on autonomy 
technology by computational complexity, as discussed in 
the previous section, Such faults are today typically 
conceptualized in terms of constraints on algorithms rather 
than cyber-security vulnerabilities; this paper aims to raise 
awareness of that gap in our understanding, 

The systems test and evaluation community has 
recognized that something is really different about 
autonomous systems, specifically, the near infinite number 
of potential system states in an intelligent, autonomous 
system renders much of existing test and evaluation 
methodology insufficient (or at worse, ineffective) for 
producing high confidence assertions of performance and 
reliability (Dahm 2010). The ideas presented here ideally 
ought to lead to enhanced test and evaluation processes, 
but we leave that to be discussed elsewhere 

In the search for novel fault modes, we are guided by 
our (admittedly imperfect) analogy to human 
psychopathology and certain philosophical considerations. 
If the computational mechanisms of intelligence are 
independent of the physical medium that supports such 
computations, then what is true of one type of intelligent 
system may also be true of another type. This is implied by 
the philosophical formulation of — machine-state 
functionalism (Putnam 1979) upon which much of 
artificial intelligence and cognitive science research is 
predicated. 


The subsections below describe potential fault modes 
that may arise in an example set of functional areas 
common to many intelligent, autonomous systems. In each 
case, we would like to understand the symptomology of 
faults and the underlying causes. Only then can we 
investigate vulnerabilities, methods of detection, isolation 
and repair. Without presenting tutorial information best 
found elsewhere, we consider potential fault modes arising 
in the processes of: 

1. Goals and Goal Generation 

2, Inference and Reasoning 

3. Planning and Execution Control 

4, Knowledge and Belief 

5. Learning 


Goals and Goal Generation 
Goals are the primal initiator of behavior in a deliberative 
autonomous system (in contrast to a reactive autonomous 
system, for example, one based on a subsumption 
architecture (Brooks 1988) which is driven more directly 
by sensory data; many autonomous systems are hybrids of 
deliberative and reactive components. In deliberative 
systems, a goal state may be completely specified, only 
partially specified, or may be in the form of a general 
preference or constraint model with “goodness” evaluated 
according to certain formulae. A wide variety of 
preference/constraint models exist, some applicable only to 
deterministic domains and others to probabilistic domains 
or where preferences must be explicitly elicted (Gelain et 
al, 2009; Dalla Pozza 2011) 

Some examples of candidate psychopathological fault 
modes related to goals that are shared with people, but not 
other non-intelligent machines, are: Disorders of Attention, 
Goal Conflict, Indifference, and Self-Motivated Behavior. 
We examine each of these in turn. 

Disorders of Attention. The pursuit of goals, including 
goal generation, goal selection, and deliberative planning, 
all require allocation of system resources. In each of these 
functions, decisions are made about how to use 
computational resources, These decision-making processes 
can be viewed fundamentally as attention management 
mechanisms (Helgason, Nivel and Thorisson 2012), 

Goal generation functions (triggered by external or 
internal information) are fundamentally _ vigilance 
mechanisms because they can divert attention. Diverting 
attention diverts the management of scarce system 
resources. In most cases, this is appropriate and exactly 
what the designers of intelligent systems intend 
(Coddington 2007; Hawes 2011). 

With respect to cyber-security, however, this suggests 
that attacking vigilance mechanisms has the potential to 
divert attention and resources away from what an 
autonomous system “ought” to concentrating upon. 
Misappropriation or diversion of scarce computing 


resources is a potential critical yulnerability of autonomous 
systems that may appear as a consequence of other types of 
faults. 

Goal Conflict, The resolution of conflicting 
requirements for achieving different goals is a fundamental 
component of all AI planning and scheduling algorithms. 
There are many such planning algorithms, and equally 
many ways to resolve goal conflicts. It is important to 
remember that the guaranteed detection of goal conflicts 
during the planning process is computationally intractable. 
Heuristic methods are required in order to focus attention 
on likely sources of goal conflict (Luria 1987). These 
heuristics are also attention management mechanisms. 
Luria (op. cit.) provides a brief taxonomy of goal conflicts. 
Drawing from that taxonomy provides a good start towards 
identifying goal conflict-related fault modes (see Table 1 
for examples). These modes are each potential vectors for 
cyber-attack by an adversary with the capability to 
artificially induce the conditions that enable a type of goal 
conflict 


TYPE OF CONFLICT DESCRIPTION 


Conflict between explicit goal and 


Compromised Intent 
P default policy or implicit intent. 


Unverified knowledge of the 


Violated Defaults 
fi values of default preconditions. 
Int, Plan used in a novel situation with 
Unintended Effects 
un-modeled direct interactions, 
Human agent asserts that a conflict 
Expressed Conflict i 


exists, with or without explanation, 


Effects of plan execution result in 
an unrelated conflict (side effect), 
e.g. due to insufficient causal 
‘model fidelity, inference horizon 
etc, Ifthe effects are non-linear, a 
cascade is possible. 


Effects Cascade 


Table 1: Example Types of Goal Conflicts: 


Consider just one of the many sources of goal conflicts 
that are known: Compromised Intent. This type of goal 
conflict occurs when achievement of a goal conflicts with 
default policy or intent, It may occur because (1) a causal 
interaction is not modeled, or; (2) an inference chain is too 
long to find the conflict (as in a search with a bounded 
horizon), or; (3) unknown, explicit or implicit priorities, or 
other conditions that enable the relaxation of constraints. I 
would be greatly surprised if a reader familiar with AI 
planning systems has not seen this type of conflict. 

There is another reason it seems familiar. Return to our 
(fictional) example in the introduction, HAL 9000, of the 
movie “2001: A Space Odyssey.” Recall that the super 
secret, highest priority mission goal given to HAL is to 
investigate the monolith at Jupiter. This explicit goal 


comes into conflict, later in the mission, with the default 
policy of protecting the lives of the crew. This is just one 
of many types of potential goal conflicts that may not be 
detected before actual execution of a plan. Skipping over 
the drama of the movie, we discover that HAL chooses to 
resolve this goal conflict by killing the crew. The 
hypothetical mechanism is relatively easy to discern: a 
relaxation of a constraining default policy (crew safety) in 
order to achieve a high priority goal (investigate the 
monolith). The constraint relaxation is enabled by HAL’s 
certainty that he can complete the secret mission without 
the aid of the crew (this is also a failure of ethical 
reasoning, discussed later). In humans, unresolved goal 
conflict is a source of significant mental distress (Mansell 
2005). Similarly, resolution of goal conflict is often (but 
not always) an imperative in autonomous systems. 

Indifference. This type of fault is a milder form of goal 
conflict that can result from an intelligent system 
concluding that (1) a human-provided goal has insufficient 
priority relative to other goals, or (2) the system itself does 
not possess the competence to achieve a goal, or (3) the 
goal is irrelevant, The consequence of goal conflict 
resolution is the human-provided goal is dropped and no 
action occurs to achieve the goal. In human terms, this 
condition is described as apathy. 

Self-Motivated Behavior. Autonomy necessarily implies 
a degree of choice of actions, whether they originate from 
externally provided goals or goals internally generated. In 
the latter case, the addition of autonomic processes to a 
system (e.g., for health maintenance, energy management 
and so forth) can result in goals that conflict with on-going 
activities. The examples we see today, such as a robot 
vacuum cleaner stopping to recharge itself, are expected 
behaviors and not of interest. However, as intelligent 
robots are deployed into dangerous situations, such as 
urban rescue or a battlefield, their autonomic functions are 
likely to expand to include sel/-preservation as a default 
autonomic function, Consider the possibility that an 
undesirable machine behavior (perhaps as a result of 
another fault and/or subversion) results in a shutdown 
command. Due to a conflict with the internally generated 
goal of self-preservation, the directive to shutdown could 
be ignored in certain situation-driven conflict resolutions. 

It is important to note that both Indifference and Self- 
Motivated Behavior may have the appearance of self- 
awareness. Yet, the information and goal conflict 
resolution processes are localized within the intelligent 
system, The appearance of self-awareness is an emergent 
psychological effect (Lewis et. al. 2011); actual self 
awareness is not required, 

Inference and Reasoning 

There are several important sources of potential faults in 

the area of inference and reasoning that require further 


study. Some are familiar to many of us from long coding 
and test sessions with intelligent systems, others are 
speculative possibilities that may arise in future systems. 

Invalid Logic. Often termed “fallacies of inference”, 
there are many forms of invalid logic that humans 
demonstrate. As yet, intelligent systems only suffer from a 
few. One of these cases is when “true” data results in a 
false answer as a result of a failure of inductive reasoning; 
for example, when an intelligent system is near the edges 
of its competence. As a consequence, insufficient previous 
experience (e.g., manifested as an incorrect probability 
distribution) results in over-weighted confidence for 
derived conclusions. This leads to the possibility that new 
data becomes marginalized or discarded rather than serving 
in a corrective function. This is an example of the classic 
“over-generalization” problem in machine learning, where 
important features that discriminate situations are ignored. 

The Fallacy Fallacy. This fault mode is complementary 
to Invalid Logic. Knowledge bases are inherently 
incomplete, likely to contain errors, and subject to many 
other limitations. One potential consequence is that a 
conclusion is dismissed because the logic used to derive 
the conclusion is faulty or incomplete, i.e., there is no 
inference chain to the conclusion that can be constructed 
from the knowledge and data given (or as a result of a 
bounded search horizon, as discussed earlier). If the 
argument contains a fallacy, i.e., invalid logic, then it is the 
argument that must be dismissed. The failure to construct 
an inference chain does not prove that the conclusion is 
incorrect, only that it cannot be proved with what is 
known, The conclusion may in fact be correct. Few, if any, 
extent intelligent systems respect this distinction; it is a 
defect of reasoning that unfortunately shared by many 
people as well. 

Solipsism. One of the dangers of the AI craft of applied 
epistemology arises from the quest to manage uncertainty. 
This has the potential to result in a sort of logical 
minimalism where sense data is subject to extreme 
skepticism and as a result, internally derived inferences 
may accrue more confidence than those based on empirical 
observations. In a sense, this is the robot equivalent of the 
human psychopathological condition of detachment from 
reality. The danger arises when solipsism undercuts 
externally imposed policy-guided constraints on behavior 
by authority. 


Planning and Execution Control 

There are a great many faults that can arise during the 
planning and execution control processes, including many 
of those related to goals as we have discussed above. 
Planning is essentially a search problem with surprising 
complexity that often requires exponential computation, 
i.e., is NP-hard (Chapman 1987; Hendler et. al. 1990) One 
of the most important potential fault modes of planning 


and execution control has only been recognized in the past 
few years: failures of ethical behavior (Arkin 2012; 
Bringsjord and Clark 2012). 

Ethical reasoning may fail due to bounded rationality. 
Depending on the circumstances, knowledge and analysis 
of the situation and actors may not be sufficient to reason 
about duty to ethical concerns. It is also true that creating 
an ethical code that is complete, unambiguous and can be 
applied correctly in every situation is notoriously difficult 
(Bringsjord 2006). Many possible algorithms to remedy 
this have been discussed (and fewer implemented), such as 
“ethics governors” (an execution monitoring system with 
veto power; essentially equivalent to the proverbial 
“restraining bolt"). Other theorists suggest that moral 
behavior will arise not from externally imposed 
constraints, but only from internally generated self 
regulation of behavior based on the utilitarian concerns of 
interacting with humans in a social world. 

A discussion of ethical behavior by machines is not 
complete without a consideration of deception, defined for 
our purposes here as a “false communication that tends to 
benefit the communicator” (Bond and Robinson 1988). For 
reasons of space, a complete review is not possible here. 
With respect to our concerns regarding fault modes and 
cyber-security, it is important to note that deception by an 
autonomous, intelligent system can arise naturally as an 
adaptive response to certain situational conditions 
(Floreano 2007; Mitri 2009), as a strategic choice, e.g., in 
warfare (Wagner and Arkin 2009, 2011), or as a relatively 
innocuous aspect of human-robot social interaction (Pearce 
et. al, 2014). This raises the question of how to tell the 
difference between a mistake (due to a failure or limitation) 
and an outright lie by an intelligent system. 

While there is much attention to policy-constrained 
behavior (Uszok et. al. 2008), The fact remains that today 
we cannot guarantee that the behavior of a sufficiently 
autonomous intelligent system will neces: 
explicitly stated policies, including ethical rules. The 
consequences might be relatively minor or they might be 
as major as the HAL 9000 goal conflict resolution example 
discussed earlier. 

Emergent Behavior. As the technology of multi-agent 
systems has matured, the phenomenon known as emergent 
behavior has been observed, i., behavior that is not 
attributed to any individual agent, but is a global outcome 
of agent coordination (Li et. al. 2006). Emergent behavior 
may or may not represent a fault condition, The flocking 
behavior of birds is emergent and represents an important 
positive survival trait, On the other hand, stop and go 
traffic and traffic jams are emergent behaviors that 
enormously degrade the performance of traffic sys‘ 

‘As yet, no generalizable methods exist for predicting 
emergent behavior in multi-agent systems, or their 
“goodness”, in part because the task is computationally 


intractable even for very simple agents with restricted 
behavioral repertoire and restricted _inter-agent 
communication topology. Emergent behavior cannot be 
predicted by analysis at any level other than the system as 
a whole. The best that can be done is to measure certain 
trends in system-wide behavior that may lead to 
predictability (Gatti et. al, 2008; Pais 2012). 

‘A fault mode worthy of study is the possibility that an 
agent in a multi-agent system is able to assert its behavior 
on other agents in a way that triggers emergent effects 
(Lewis et. al. 2011). Two simple examples, similar in 
nature, are crowd behavior in humans and insect swarming. 
To the extent that an agent suffers some other fault, or is 
suborned, it may trigger undesirable emergent behaviors in 
the system as a whole, Despite the rush to implement 
multi-agent systems for important and critical applications 
in health, finance, transportation, defense and other 
domains, we simply do not yet have an understanding of 
fault modes that are likely to occur due to emergent 
behavior. 


Learning, Knowledge and Belief 

This category of potential fault modes is quite broad and 
truly deserves more attention than this short paper can 
afford. Nevertheless, it is important to highlight a few fault 
modes that may be quite common to intelligent, 
autonomous systems. These all arise from the autonomous 
processes involved in creating, maintaining, and adapting 
what an intelligent system believes to be true, 

The most glaring example of this type of fault mode is 
faulty or absent truth maintenance, i.e., the ability to retract 
assertions previously thought to be true which are now 
rendered invalid by new information (defeasibility) 
Formally, this is a property of first order logical 
“monotonic” systems. The use of monotonic inference is 
not in itself a fault. If previous inferred assertions do not 
play a role in future reasoning, they are effectively 
discarded if not explicitly falsified when contradictory 
information is obtained. For example, a credit card fraud 
detection system might depend exclusively on salient 
features in a single case of use of the card. The fact that a 
previous use of the card was valid does not automatically 
validate a new use of the card. First order logic is common 
in many applications. 

However, intelligent systems that build models of the 
world, actors, situations, and so forth via machine learning 
must use non-monotonic reasoning (second order or higher 
logics) to achieve defeasible inference. Given the 
uncertainty inherent in a dynamic and uncertain world, 
defeasibility can be a difficult process because it requires 
weighing the evidentiary force of new data against 
previously derived probative assertions. In a sense, 
skepticism must balance a rush to learn or “correct” 
previous beliefs. 


This is where computational argumentation and. its 
contribution to persuasive technology may have an 
important role. While the topics are strongly related to 
formal logic and mathematical proof, they transcend it in 
several ways. Most important to this discussion is the 
explicit inclusion of dialog in the process of 
argumentation, often in the context of creating 
“explanations” as to why certain conclusions have been 
reached, as in intelligent decision-support systems (Bench- 
Capon et. al. 1991,2007a). In this context, argumentative 
dialog is an exchange of ideas using rhetorical methods of 
persuasion that include social methods as well as 
mathematical logic. 

Justification of belief though argumentative dialog opens 
the door to fallacious reasoning as a method of persuasion. 
“Appeal to Authority” (argumentum ad verecundiam), 
while regarded as fallacious in theories of debate, cannot 
be ruled invalid simply by noticing it in dialog ~ it requires 
a further exchange of ideas. In the absence of effective 
counter argument, by either human or machine participant, 
fallacious reasoning may be highly influential as a result of 
“practical reasoning”, i.e., an assertion is correct within the 
perspective of one of the agents involved (Bench-Capon 
and Dunne 2007b). Humans are particularly vulnerable to 
deceptive cognitive illusions that result from certain 
argumentation strategies and practical reasoning. The 
computational methods for exploiting this weakness are 
actively being explored (Clark and Bringsjord 2008), 

The cyber-security concer is that practical reasoning to 
justify belief in the presence of uncertainty opens the door 
to the possibility that an adversary could, though the 
argumentative dialog process, undermine an intelligent 
system's beliefs. This would be an even greater risk in the 
context of supervised learning with training data. 
Supervised machine learning is already known to be 
subject to a number of systematic biases, including for 
example, order bias, recency bias, and frequency bias 
Errors in causal attribution can easily result from these 
biases. 

‘A second, related cyber-security concern is the role 
practical reasoning could play in goal generation and 
planning. By undermining (or cunningly shaping) an 
autonomous, intelligent system’s beliefs, all of the goal- 
related fault modes discussed earlier could be induced 


Conclusions 


Inherent in the concept of autonomy in intelligent systems 
is the ability to make choices about what to do and how to 
do it, These are fundamentally mechanisms for managing 
attention and vigilance. In this paper, we have examined 
some of the components of intelligent systems that support 
autonomy and discussed a selection of potential fault 


modes. Some of these fault modes require a degree of 
meta-cognition that, while not yet realized in autonomous 
systems, is an active area of research. 

It is possible that some or all of these fault modes can be 
induced, and as a consequence, there now exist new and 
unique cyber-security concerns surrounding autonomous 
systems that must be explored, It is therefore incumbent on 
the Al research community to establish a theoretical and 
empirically substantiated foundation for cyber-security 
issues related to autonomy, with special attention to gaps in 
current knowledge. Future studies of cyber-attack 
vulnerabilities, per fault modes that are related to 
autonomy, should explore the following: 

1. Fault Modes: Are there new types of fault modes 
that can be exploited? Which fault modes are 
possible to induce, and in what manner and 
circumstance? 

Detection: How can we detect that an intelligent, 
autonomous system has been/is being subverted? 

3. Isolation: In the context of autonomous system 
faults and possible subversion, what do the 
traditional system concepts of fail safe and fail 
operational mean? 

4. Resilience and Repair; What are the proximal 
causes of the observable symptoms of autonomous 
system fault modes and how can these be mitigated? 

5. Consequences of Vulnerabilities: What are the 
consequences of deception by an autonomous, 
intelligent system (whether it has been subverted or 
not)? What is the impact of different types of fault 
modes on human reliance, trust, and performance of 
human-machine systems? 

The inspiration for this paper was the question of 
whether fictional dramatic accounts of computers/robots 
“run amok,” often described in anthropomorphic terms, 
have the potential to actually occur either with existing 
technology or technology that can be reasonably foreseen 
on the horizon, Some, but not all, of these faults and 
vulnerabilities have useful analogies to psychopathologies 
of the human mind. The development of a theory of 
“psychopathology of intelligent machines” has the 
potential to provide insight into aspects of computational 
intelligence just as studies of human mental disorders 
provide insight into the functioning of the brain. The 
methodology that remains to be developed will guide us 
towards computational approaches that design out, work 
around or otherwise mitigate these failure modes and 
potential cyber-security risks. Ultimately, the real payoff is 
the opportunity to increase the trustworthiness of machine 
intelligence; in the absence of justifiable trust, the full 
potential of autonomy technology will not be realized. 
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Abstract 


We present an emerging research project in our laboratory 
to extend ambient intelligence (Aml) by what we refer to as 
“extreme personalization” meaning that an instance of 
ambient intelligence is focused on one or at most a few 
individuals over a very long period of time. Over a lifetime 
of co-activity, it senses and adapts to a person’s preferences 
and experiences, and crucially, his or her (changing) special 
needs; needs that differ significantly from the normal 
baseline. We refer to our agent-based cyber-physical system 
as Ambient Personal Environment eXperiment (APEX). It 
aims to serve as a Companion, a Coach, and a Caregiver: 
crucial support for individuals with mental, physical, and 
age-related disabilities and those other people who help 
them. We propose that an instance of APEX, interacting 
socially with each of these people, is both a social actor as 
well asa cyher-human prosthetic device, APEX is an 
ambitious integration of multiple technologies from 
Artificial Intelligence (AI) and other disciplines. Its 
successful development can be viewed as a grand challenge 
for Al. We discuss in this paper three research thrusts that 
lead toward our vision: robust intelligent agents, 
semantically rich human-machine interaction, and reasoning 
from comprehensive multi-modal behavior data. 


Introduction 


Ambient intelligence (Aml) extends and combines earlier 
paradigms of pervasive computing with sensor networks, 
human-centered interfaces, mobility, Artificial Intelligence 
(AD, robotics, intelligent agents and the “Internet of 
Things.” The concept is compelling as it promises to 
deliver an integrated computing, device and networking 
infrastructure that provides services while remaining 
largely hidden from the view of users. 
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The research themes in our laboratory seek to extend 
Aml by what we refer to as “extreme personalization”, 
meaning that a given instance of APEX is focused on at 
most a few individuals, e.g. client, caregiver, and 
physician, over a very long period of time. Over a lifetime 
of co-activity, it senses and adapts to each person’s 
preferences and experiences, and crucially, his or her 
(changing) special needs. We refer to this agent-based 
cyber-physical system as Ambient Personal Environment 
eXperiment (APEX). It serves as a Companion, a Coach, 
and a surrogate Caregiver to the client: crucial support 
roles for individuals with mental, physical, and age-related 
disabilities. It supports the primary caregiver, automating 
some tasks, advising, and facilitating interaction with the 
client. It provides the client’s remote physician with an 
“extra set of eyes” for monitoring the client’s progress and 
by being alert for anomalies requiring medical attention, 
We propose that an instance of APEX, interacting socially 
with each of these individuals, is both a social actor as 
well as a cyber-human prosthetic device (Hamilton 2001). 

APEX must learn, adapt and perform in a natural 
environment that is rich with features, many of which are 
usually irrelevant or at least uncertain, To achieve this, 
APEX depends upon successful integration of multiple 
technologies from artificial intelligence, human-centered 
design, cognitive science, computational linguistics, 
human-machine interaction, and robotics. As such, it 
represents a major stretch goal that is likely to drive each 
area in some new directions. While incremental results will 
certainly be useful, the achievement of APEX surely 
represents an interesting grand challenge with high social 
value. 

In the sections below, we present a brief overview of 
selected functional requirements and three research thrusts 
that lead toward our vision for APEX: (1) robust 
intelligent agents; (2) semantically rich human-machine 


interaction, and; (3) reasoning from comprehensive multi- 
modal behavior data. Each poses unique challenges, yet the 
technologies that support each area form a synergistic 
combination that we contend will lead us towards 
generalizable solutions. 

We conclude with a discussion of related work, the 
broader impact of this type of system for AI and related 
research, and the potential social benefits as well as 
emerging risks to privacy and security. 


Ambient Personal Environment 


Our vision of APEX as a cyber-human prosthesis for 
persons with mental, physical and age-related disabilities is 
driven by two overarching functional requirements: 

+ The ability to sense, interpret_and change a 
person's environment (e.g., physical objects, 
enclosed spaces, ambient attributes) in the context 
of delivering specific health-related support 
services. 

+ The ability, over potentially a lifetime of co- 
activity, to learn and adapt to an individual’s 
changing special needs that arise from mental, 
physical, and age-related disabilities. 

Additionally, since APEX must interact with multiple 
people, it must fill a niche that is complementary to the 
other actors in a disabled person’s life. This requires: 

+ The ability to function as an intelligent actor-agent 
in the role of a Companion, a Caregiver, and a 
Coach, aiding and easing the burden to people in 
these traditional roles. 

Therefore, it is critically important for the success of 
APEX not only that we address user needs that are 
common to the community of people with cognitive, 
physical, and age-related disabilities, but that we also 
consider the needs of the people who are part of the the 
APEX clients’ lives. We envision a long-term relationship 
with differing levels of disability and sickness, though here 
we will refer to periods where such help is most needed.. It 
is probably more correct to say that APEX becomes 
adapted to the entire system—the people, activities, and 
environment as a whole, centered around the person with 
the disabilit 

While space precludes an extended discussion of the 
needs particular to each of these roles, there are several 
requirements that should be highlighted as they are broadly 
common to research on ambient assisted living. 


Individual Client Needs 

The practice of nursing observation of patients in a hospital 
setting provides some important topics to consider in AmI 
research, In current practice, nursing observation is 
necessarily intrusive. Many patients have negative 


reactions associated with a high level of intrusiveness. 
With lower levels of intrusion, patients report positive 
effects of observation including a sense of support. 
However, these benefits are negated if patients feel 
observers lack empathy or seem remote. They also react 
negatively if they feel they are not given sufficient 
information about the purpose of observation or a medical 
process to be provided by the nurse. Page (2006) provides 
an excellent review of relevant studies as well as citations 
toa rich set of original sources. 

Therefore, new studies are required in order to help 
parameterize the idea of “an optimal sense of 
intrusiveness,” that is, intrusiveness that elicits the positive 
affect of support without the negative affect associated with 
observation, Furthermore, APEX must simulate empathy 
based on modeling the client and use that to guide 
informative interactions (Bee et. al, 2010; Kearns et. al 
2014). 


Caregiver Needs 


Many clients haye a human caregiver and it is important to 
recognize the importance of augmenting rather than 
replacing them, even as APEX provides essential help to 
the client that might otherwise fall to the caregiver. APEX 
must adapt and be functional with respect to. the 
caregiver's role, preferences, and intentions, serving as the 
caregiver's agent even when the caregiver is not present. 
Augmentation, not replacement, is a common requirement 
for many applications of automation and this case is no 
different, Our approach is to position APEX as an aide to 
the caregiver, for example, by assisting in communication, 
reminding, observation, and so forth. Our recent studies, 
discussed below, with veterans who have suffered 
Traumatic Brain Injuries (TBI), have highlighted the 
potential benefits of mediating communication between 
client and caregiver using Companion agents (Wilks et, al 
2014). 


Primary Physician Needs 


Physicians must reconcile the dual need to provide 
excellent health care while avoiding excessive office or in- 
patient visits. Once a patient is discharged, e.g., from a 
Veterans Administration hospital brain injury rehabilitation 
unit, the primary physician’s focus is on maintenance of 
stable day-to-day health, rehabilitative progress to the 
extent it is possible, and remaining alert to any signal that a 
client's condition requires immediate or near-term 
attention, These are three areas where APEX may usefully 
assist physicians, The early detection of clinically 
meaningful anomalies in client behavior would be very 
valuable to physicians and this is one area we are pursuing, 
building on previous work by our collaborators at the 


Veterans Administration Polytwauma Unit in Tampa, 
discussed below (Kearns, Nams and Fozard 2010). 


Technical Approach 


Our general approach is to explore computational 
models that integrate core Al components with intelligent 
agent architectures to enable robust, trustworthy adaptive 
autonomy in the context of long-duration human-machine 
joint activity. In so doing, we will necessarily push the 
limits of core Al algorithms for natural language, multi- 
modal social interaction, theory of mind, and more. As we 
explore these models, it is apparent that the work will 
benefit greatly from multi-agent model-based discrete 
event simulation for guiding the design and tuning of 
complex cyber-physical intelligent agents, experimental 
design and the assimilation of vast quantities of sensor data 
to models for analysis and theory development (Clancey et. 
al, 1998, 2007). Our formulation of specific studies will be 
guided by analysis of parameterized exploratory 
simulations of use-case scenarios that we are now 
developing, 

APEX includes a physical laboratory, built inside to 
resemble a small home. Previous work in this shared 
facility has focused on learning-by-observation for cooking 
tasks in a high quality kitchen. We are building out 
additional mock rooms where the walls and ceiling contain 
our multiple sensors and interactive devices including 
touch screens, structured lighting, motion capture, and 
more. Completion of the lab build depends on the timing 
and availability of appropriate funding; at the time of this 
writing we can only report, “in progress.” 


Robust Intelligent Agents 


An instantiation of APEX is an autonomous, intelligent, 
and social agent (at times personified and/or embodied as 
discussed elsewhere in this paper) that takes the role of a 
life-long companion, providing highly personalized and a 
dynamic, ever-changing degree of assistance and support 
for healthy assisted living. In this way, APEX differs 
substantial from other agent technology applied to 
healthcare (Isern, Sanchez, and Moreno 2010). The 
capability to predict, plan, and manage physical effects 
along with attention to individual behavior along 
psychological and social dimensions of the disabled 
individual, caregiver and primary physician requires a high 
degree of shared awareness. This forms a basis for human- 
machine trust, a foundation of teamwork and the adaptive 
autonomy for effective human-machine joint control that 
the APEX problem domain requires (Atkinson, Clancey 
and Clark 2014), 

Successful sensing and planning along behavioral and 
psycho-social dimensions may be achieved by using 


predictive cognitive models to underlie the system’s 
“theory of mind” regarding others (Premack and Woodruff 
1978). Though it is already a challenge to model typical 
humans, for APEX the matter is complicated by the fact 
that many clients will be cognitively impaired. Some 
progress has been made toward modeling various cognitive 
impairments such as Autism and Alzheimer’s disease using 
existing cognitive architectures (Matessa 2008; Serna, 
Pigot, and Rialle 2007), but the development of atypical 
cognitive models remains difficult, For clients with TBI, 
the applicability of existing cognitive architectures is 
unclear particularly because of the range of function that 
people with TBI can have from very low (with no initiative 
or memory) to relatively high. The nature of TBI is such 
that the individual's impairments are at once both profound 
and highly unique, which serves to undermine common 
modeling assumptions and architectural commitments 
regarding cognitive processes and capabilities. 

We envision APEX to have access to real-time and 
historical observation data; therefore, our approach to 
client modeling is to exploit machine leaning (e.g., 
statistical, inductive techniques) where possible. This 
includes using the methodologies of behavior analysis 
(Cooper, Heron, Heward 2007) to develop predictive 
models of client behavior that is contingent on objects and 
events in the environment and the client’s history therewith 
(observables as opposed to invisible cognitive processes).! 

Our data-driven approach to the creation and 
maintenance of client models naturally allows 
accommodation, integration, and adaptation to learned user 
preferences, observed long-term trends (such as recovery 
or disease progression), and event-triggered short-term 
phase changes (such as the temporary effects of a recently 
taken medication). Finally, while the purpose of APEX is 
not therapeutic, at times assistance and joint action may 
require APEX to motivate or gain compliance from the 
client. Behavior analysis provides an appropriate 
methodology and ethical framework for such 
manipulations. 

‘As observers and aggregators of various forms of 
personal information (e,g., behavioral, medical), there are 
numerous privacy and security concerns with computer 
systems like APEX (e.g., safeguarding against accidental, 
illegal, or malicious compromise of data; means for 
individuals to exercise control over their personal data). 
Moreover, the autonomy imputed to intelligent agents 
brings with it issues of ethics and whether such agents are 
or ought to be ethically bound, For example, imagine that 
a client confides in APEX that he or she is contemplating 


"This is not to say that we are abandoning main stream cognitive 
modeling; we are trying to forge a happy marriage of cognitive and 
behaviorist methods, Sustained discussion of the relative merits of each is 
beyond the scope of this paper. 


suicide, Does APEX have a duty, ethically, legally, and/or 
morally (Wilks and Ballim 1990), to report the client's 
statement to others ~ and if so, is APEX liable if, in fact, it 
was only a case of gallows humor? Do we want APEX to 
have privileged confidentiality like an attorney or do we 
want it to be a mandatory reporter like other medical 
professionals? And how should privacy, data ownership, 
and ethical duties be weighed against equality in a 
convalescent or group home setting? Will APEX behave as 
a loyal friend, and be dedicated first and foremost to the 
client? Or will it appear as an agent representing the 
caregiver and physician? How are such dual purposes 
reconciled to establish the trust of all the players? 
Innovative home-based services like APEX provide a new 
impetus for academic and social discussion of such ethical 
concerns and risks, which are far from resolved. 


Semantically Rich Human-Machine Interaction 


The disabled and/or aging users whom we are targeting 
with APEX pose a variety of unique challenges for human- 
machine interaction. Cognitive disabilities may impair both 
interpretation and generation of language; physical 
disabilities may impair one or more signal channels, e.g., 
vision, speech, gesture; general aging may affect 
communication tempo and other attributes of interaction, 

Our focus is on the use of multiple modalities for 
human-APEX interaction. In any given interactions, 
modalities will dynamically adjust in composition and 
manner of use (e.g., signals and protocols) based on 
context, client capability, and other communication 
exigencies of the moment (e.g., urgency to take medication 
on time). 

Interaction must address the complexity of human- 
machine trust, especially when APEX must behave in a 
dominant manner to coach and guide behavior. Many 
veterans with TBI, for example, have a strong distrust of 
authority. Other users could simply fail to comply because 
they are skeptical of APEX’s competence, or feel it is 
“hiding something.” Compliance of APEX with the 
constraints and demands of human social interaction is 
paramount (Atkinson and Clark 2013). 

To address the trust-related concerns as well as the 
possibilities of providing (1) a unique modality for non- 
verbal interaction and,(2) active physical assistance, we are 
investigating the use of humanoid robots as an in-home 
vatar” for APEX, An embodied avatar, much like 
Embodied Conversational Agents (ECA), will evoke 
human social expectations and interaction very effectively 
(Schaefer, Billings and Hancock 2012) with quantifiable 
risks and benefits (Dorneich 2012). For a cogent review of 
research, application, and evaluation of embodied 
conversational agents, see Cassell (2000). Recent research 
projects in this domain include SEMAINE (Schroder, 


2010), VHTookit (Hartholt et al. 2013), and Companions 
(Wilks et al. 2011). 

In the near-term we are planning to investigate the 
proposition that a humanoid avatar, compared to a 
disembodied visual interface, will perform better in 
guiding client navigation in the home (e.g., to the medicine 
cabinet) and ensuring compliance with a pre-determined 
schedule of activities. A very small mobile robot would 
likely be sufficient for this purpose but would be incapable 
of performing physical labor. 

A more robust robotic system would be capable of 
providing direct physical assistance to the client, such as 
helping in the kitchen, finding and providing the television 
remote control, ting in standing, or dispensing 
medication (Figure 1). Mobility could be provided on the 
ground (¢.g., wheels or legs) or via an overhead rail 


Figure 1: A Robot Assisting with Medication 


A significant component of our rich human-machine 
interaction is that of automatic speech processing, with an 
emphasis on understanding of, and adaptation to, speech 
that is impaired. Borrowing from the field of machine 
translation (Dorr 1993), we adopt a paradigm in which the 
notion of divergence is central. 

To illustrate the concept of divergence across languages, 
consider three properties (vocabulary, pronunciation, and 
syntactic structure) coupled with the differences across 
these for four languages: Spanish, Portuguese, English, and 
Chinese. We may consider a language to be similar to 
another language in “vocabulary” if there is a shared 
orthography, in “syntax” if the grammars are the same, and 
in “pronunciation” if they contain similar phonological 
forms. The most similar language pair of these four (aside 
from the language to itself) is Spanish-Portuguese, which 
shares all three features. The most radically divergence 
pair in is Spanish-Chinese, where there no similarities are 
‘ociated with any of these three features. 

We apply this same notion of divergence to the problem 
of “speech functioning,” constraining our language pair to 
asymptomatic English speech compared to impaired 
English speech. In this case, the divergence properties to 
be studied are articulatory and disfluency patterns. We 
develop and apply techniques for detecting such 


ass 


divergences and leverage these to enable adaptive 
automatic speech recognition. The goal is to adapt to both 
deterioration and improvement in speech, within the same 
person, over time. For example, in Amyotrophic Lateral 
Scleroris, speech is likely to become more impaired, 
whereas with Traumatic Brain Injury, the speech is likely 
to become less impaired. 

The closest speech processing study to the divergence 
approach described above is by Biadsy et al. (2011), who 
investigated the variation of speech properties under 
intoxicated and sober conditions. However, this earlier 
work was applied to the detection of intoxication (vs. 
sobriety), not the degree of intoxication, Rudzicz et al. 
(2014) employed another approach for recognizing 
impaired speech to answer a similar yes/no question 
(Alzheimer's vs. no Alzheimer’s), Although the notion of 
‘degree” was not the focus of these earlier studies, we 
leverage the incidental but significant discovery that 
pronunciation varies systematically within categories of 
speech impairment, This discovery is critical for 
correlating the divergence from a baseline English and 
provides a foundation for adapting speech recognition 
technology to varying degrees of impairment. 

In the overarching APEX framework, the studies above 
are significantly enhanced beyond individual speech 
recognition experiments, in three ways: 

+ We benefit from the potential for embedding this 
technology into the three paradigms mentioned 
above (companions, humanoid avatar, and robotic 
systems) to enable conversations with a computer. 

+ We leverage the paradigms above to investigate 
interactive dialog that includes informal language 
understanding, in the face of disfluencies such as 
filled pauses (ui), repeated terms (/-/-I know), and 
repair terms (she—/ mean—he). 

+ We are able to investigate pragmatic interpretation 
of language and action, thus undertaking intention 
recognition, Sensor input (visual, tactile, etc.) 
enables the understanding of utterances that are 
otherwise uninterpretable due to speech impairment, 
e.g., Fill it with rockbee may be understood with 
gesture toward a coffee cup may be understood as 
Fill it with coffee. 


Reasoning from Comprehensive Multi-modal 
Behavioral Data 


A major challenge for APEX, indeed for many adaptive 
intelligent agents that are focused on human-computer 
interaction, is collecting, aggregating and analyzing very 
large amounts of longitudinal data from heterogeneous 
sensors. APEX requires a symbolic, _ temporal 
representation of “now,” that is, what happened previously 
that led to the present situation, and what is likely to 


happen in the future under various hypothetical conditions. 
From such a basis, APEX must maintain situational 
interpret behavior, and infer intent. This 
capability is a fundamental basis for real-time reasoning 
about human behavior in the context of environment 
dynamics, It provides essential support for decision 
making and closed-loop physical automation, Finally, 
experiential knowledge is fodder for non-real time 
reflection that leads ultimately to the machine learning and 
adaptation we believe is required. 

We will use automated coding of multi-modal 
behavioral data to achieve situational awareness of the 
client. The field of behavioral signal processing (BSP) has 
used automation to model abstract human behaviors in 
relevant, realistic scenarios, mitigating previous manual 
behavioral sciences coding schemes. An overview of 
automated methods that are maturing rapidly includes 
discussion of social cues, affect, and emotion (Black et, al 
2011). 

An early application of BSP technology is currently 
fielded as a “Smart Home’ by the Tampa VA Hospital 
Polytrauma Center (Jaziewiez et. al. 2011). This Smart 
Home continuously collects and analyzes client location 
and orientation data, as well as every interaction of clients 
with clinical and medical staff, Early data mining analyses 
using a BSP method called Fractal D have provided insight 
into gait and walking behavior (e.g., wandering) that would 
otherwise not have been detected or quantitatively 
documented if dependent on human observation alone 
(Kearns, Nams and Fozard 2010). 

Our approach to level-one data processing of the multi- 
modal sensor data acquired by APEX will include an 
approach for well-structured behaviors (¢.g., “sit down”). 
We will generate a probabilistic template built from 
training examples based on motion-capture data, Well- 
known algorithms such as stochastic context-free 
grammars can be used to make probabilistic matches to 
such templates using limited sensor data (Abowd et. al. 
2002). 

Level-two processing will assimilate these tokenized 
situational and behavioral data to dynamic world models 
that represent the evolution of situations, intentions, 
activities, and other elements of shared awareness 
(Atkinson, Clancey and Clark 2014). Previous studies have 
shown the viability of detecting clinically relevant changes 
in behavior using this type of longitudinal sensor data and 
activity recognition algorithms (Dawadi and Cook 2014), 

Reasoning in APEX will be driven by goals that range 
from baseline policies that always constrain possibilities 
(e.g., keep the client safe) to goals that reflect physician 
general guidance extending over some period (e.g., take 
the medicine twice each day at mealtimes), to reactively 
generated goals that are a function of interaction with the 
client or care-giver, or other situational exigencies of the 


awarene: 


moment (¢.g., the stove is left on after cooking is 
complete). 


Conclusion 


In the sections above, we have presented our research and 
vision for the Ambient Personal Environment eXperiment 
(APEX). This is an exciting new project that brings 
together disparate areas of artificial intelligence and 
promises to reveal new challenges for cyber-physical 
systems and robotics in the context of ambient intelligence 
applications. 

Integrated system studies, such as APEX, require 
multidisciplinary contributions. If those studies are 
performed in a common experimental environment, such as 
the one we are constructing, it will facilitate both 
collaboration and technology integration, thereby 
increasing the chances for success in creating valuable 
system-level advances that address important individual 
and social needs. 

Our system-level approach gives us the opportunity to 
investigate challenging use cases of interest to clients, 
clinicians, and caregivers. These include _ health 
monitoring, remote health care, support for rehabilitation 
and independent living, and systems that promote and help 
ensure health through ambient persuasive technologies 

The latter is a topic area fraught with ethical concerns 
(Verbeek 2009). Persuasive technology applied in health 
care requires especially careful consideration and 
discussion of methodological and ethical factors with 
respect to informed consent and privacy. The notion of 
informed consent is very important in any care system 
embedded in a society with strong legal constraints and 
recourse such as the US. Thus, it would be of great interest 
if an APEX-like agent could also elicit informed consent 
from clients after a process of explanation and 
conversation based on deep knowledge of them (Wilks and 
Ballim 1990). That would not only economize on 
expensive professional time, but would be a genuine 
cognitive advance into an area where an automaton was 
able to make an informed judgment about a client’s mental 
state: that of understanding and consequent consent to 
procedures. 

It is our hope that this research, and those of our 
colleagues working on ambient assisted living technology, 
will eventually help meet the needs of an overburdened 
health system and an aging society. That burden is severe 
Of the two million soldiers who have served in Iraq and 
Afghanistan, as many as eight hundred thousand have 
suffered traumatic brain injuries that resulted in some level 
of cognitive impairment”, We seek to provide essential 


? Personal communication to author by a senior physician in the Tampa 
VA Polytrauma unit who is involved in VA planning for these veterans. 


technology that allows these wounded veterans and others 
with special needs to remain in their homes and participate 
in society with a high quality of life. 

Key contributions and points to remember are: 


+ Ambient intelligence (AmI) for people with cognitive, 
physical or age-related disabilities requires extreme 
personalization, adaptation significantly beyond the 
baseline of Aml for general users. 

+ Extremely personalized systems must be able to learn and 
adapt to a person's changing needs over a life-time of co- 
activity, 

+ Unique individual cognitive or physical disabilities 
require an Aml to interact flexibly with a client through 
multiple modalities whose needs cannot always be 
foreseen without actual experience. 

+ An Ami that provides support to a person with disabilities 
is both an actor-agent and, by virtue of substituting for 
lost abilities and augmenting others, a cyber-human 
prosthesis. 

+ APEX is our Am laboratory for bringing together the 
multiple disciplines and technologies required in order to 
achieve this vision. 
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ABSTRACT 

Well-justified human evaluations of autonomous robot 
trustworthiness require evidence from a variety of sources, 
including observation of robot behavior. Displays of affect by a 
robot that reflect important internal states not otherwise overtly 
visible could provide useful evidence for evaluation of robot agent 
trustworthiness. As an analogy, the human limbic system, 
sometimes described as an ancient sub-cognitive system, drives 
human display of affect in a manner that is largely independent of 
purposeful behavior arising from cognition. Such displays of 
affect and corresponding attributions of emotion provide 
important social information that aids understanding and 
prediction of human behavior. Could an “artificial limbic system” 
provide similar useful insight into a robot’s internal state? The 
value of affect signals for evaluation of robot trustworthiness 
depends on three crucial factors that require investigation: 1) 
Correlation of affective signals to trust-related, measurable 
attributes of robot agent internal state, 2) Fidelity in portrayal of 
emotion by the robot agent such that affective signals evoke 
human anthropomorphic social recognition, and 3) Correct human 
interpretation of the affective signals for justifiable modulation of 
beliefs about the robot agent. This paper discusses these three 
factors as principles to guide robotic simulation of emotion for 
increasing human ability to make reasonable assessments of robot 
trustworthiness and appropriate reliance 


Categories and Subject Descriptors 
1.2.9 [Roboties]: Operator Interfaces---Trust 


General Terms 
Algorithms, Measurement, Human Factors, Theory 


Keywords 
Affective Computing, Artificial Intelligence, Human-Robot 
Interaction, Intelligent Robots, Robot Ethies, Social Robots, Trust 


1. INTRODUCTION 

This research investigates methods for autonomous agents to 
foster, manage, and maintain appropriate social trust relationships 
with human partners when engaged in joint, mutually 
interdependent activities. Non-verbal behaviors ‘serve important 
functions in coordination and regulation of joint activity [6]. At 
any given moment, they provide team members with insight into 
the type and state of interaction, the state of interdependency 
(including changes in dominance and control authority), and 
provide information that aids inference about the internal states of 
team members. Gauging belief, disposition and intention are 
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important for prediction of future behavior and the evaluation of 
risk versus benefit of delegation. A robot's ability to fluently 
interact socially and build a trust relationship depends on the 
robot’s ability to help humans understand it, in part through non- 
verbal behavior. Affect-related signals are among the non-verbal 
behaviors that influence human trust [3,7] 


Humans appear to have an innate ability to evaluate the meaning 
of non-verbal emotive displays to judge trustworthiness in other 
humans. This process of intuiting internal state from emotional 
signals works because humans are similar along various 
dimensions of commonality. Our natural expression and 
understanding of non-verbal signals is one such function of 
common biological heritage. This commonality is based in the 
evolutionary development of sub-cognitive neural circuits 
(paleo-circuits”) and pathways that operate for the greatest part 
independently of purposive control. These circuits link bodily 
arousal centers, emotion centers and the motor areas of brain and 
nervous system with the muscles required for display of non- 
verbal signals [10]. Reasonable evaluation of trust arises from the 
human ability to recognize these signals and (along with other 
information), construct a set of beliefs, and make inferences about 
the internal state of other agents. Realizing the value of simulated 
robot emotion for human evaluation of trustworthiness requires 
that robots correctly evoke this innate human ability 


2. GUIDELINES FOR ROBOT EMOTION 

A robot, even with a simple morphology and electromechanical 
capability, has the potential to portray a rich repertoire of non- 
verbal behaviors that have familiar social meaning for humans. To 
analyze the potential role of these behaviors in engendering 
human-robot trust, we use the idea of a “Human Social Interface” 
[2]. A good interface specification defines channels, signals, and 
protocols that result in specific state changes among systems 
using that interface. Effective use of the Human Social Interface 
to reveal robot internal state for evaluation of trustworthiness 
using non-verbal displays requires: 


1, Reliable Signals: Robot signals, in whatever modality 
is appropriate, must be reliably correlated with the 
internal, trust-related attributes of robot state 

2. Portrayal Fidelity: Robot signals must be portrayed 
with “sufficient fidelity to evoke human 
anthropomorphic social recognition. 

3. Correct Interpretation: Robot signals must be 
interpreted such that modulations of human beliefs 
about the robot are well justified. 


2.1 Reliability of Robot Affect Signals 

Human cognitive and emotional evaluation of the trustworthiness 
of another person is based on information that may arise from a 
broad array of sources, including direct interaction, Emotional 
judgment of trust in the moment may rely heavily on non-verbal 
signals since emotional expressivity can act as a marker for 


cooperative behavior or trustworthiness [3]. When cognitive 
information and perceived affect are discordant, deception is seen 
as a possibility [4]. Humans expect non-verbal signals; consistent 
with social anthropomorphism, we conjecture that this expectation 
carries over to robots. Indeed, when robotic empathic responses 
are absent or inappropriate, trust decreases. [5,8]. 


‘A robot's extemal behavior should reflect its internal state. 
Generating correct robotic affect signals in the absence of 
common human biological heritage requires that we design robot 
non-verbal behaviors such that they reliably reflect those aspects 
of intemal robot state that are indicative of trustworthiness. There 
are many such aspects. For example, human trust requires a 
positive evaluation of competence and the ability to deliver. A 
robot behavioral display linked to its state in this case would be a 
function of the robot's relevant knowledge (of various types), 
relevant experience (episodic memory) and the applicability of the 
robot's knowledge to the present situation, An intemal, non- 
deliberative assessment mechanism (e.g., as in the human limbic 
system) of adequacy for an assignment would trigger a positive 
valence behavioral display, for example, portraying a “can do” 
attitude. 


2.2 Fidelity of Portrayal 
Readability of robot non-verbal signals is crucial. Readability is 
the characteristic of a non-verbal signal that enables it to be 
recognized as such and correctly interpreted. The ability to evoke 
human anthropomorphic recognition of a non-verbal signal and 
correct interpretation depends on situational relevance and on 
sufficiently accurate performance, ice, fidelity, of the non-verbal 
behavior by the robot. 


Continuing with the example of portrayal of a “can do” attitude, 
there are a number of signals on various channels that may be 
transmitted individually or in combination (as long as the 
combination is “natural,” ie., judged compliant with human 
expectations). See Table 1 


Table 1. Example Portrayal of a “Can Do” Attitude 


Channel Signal Protocol 
Gas (Redirection | LOOK direcily at valid objects ie 
context 
Face Expression | Relaxed and friendly 
cal Position | Tied up with chin siightly 
pushed forward 
Torso Posture | Relaxed 
P Rel, Angular | Move from neutral to angle-in 
TOMeRUCS: Position | orientation (lean fwd or to side) 


2.3 Justified Modulation of Beliefs 

Human interpretation of non-verbal signals depends upon 
cognitive and emotional processes, and on context, including a 
primary appraisal of valence, and secondary appraisals. that 
include perception of certainty (of a situation), required attention 
and effort, and control over outcomes (either self or exogenous 
factors). The correct attribution of emotion and inferences about 
the characteristics and intemal state of another agent require 
multiple signals, experience and a degree of skill (often called 
“empathy”). Ambiguity, contradiction or poorly executed signals 
are either not readable or lead to erroneous conclusions. 


We know from previous studies that certain non-verbal cues are 
predictive of human distrust [8,9]. While these studies 


acknowledge the need for a robot to use such cues to elicit trust, 
they focused on robotic interpretation of human cues. Further 
research is needed on human interpretation of robotic cues. In our 
model, non-verbal behaviors, such as those found in the studies 
cited above, influence trust between team members by 
contributing to the attribution of certain individual qualiti 
required for trust (e.g., competence, predictability, openness, risk). 
This is a reciprocal process requiring studies that combine both 
human and robotic interpretation of non-verbal cues 


3. INSIGHT OR ILLUSION? 

The human social interface must not become a means for 
manipulation by robotic “trust inducing” behavior that has no 
truth in the robot's actual state, that is, the robot should not simply 
mimic human behavior. This could easily become dangerous and 
deceptive. Our insistence on the reliability of robot affect signals 
is therefore essential for honest social behavior by robots. 
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EXTENDED ABSTRACT 


The foundation of teamwork is well-calibrated mutual 
trust among team members. The goal of our research is to 
enable trust for appropriate reliance and interdependency in 
teams composed of humans and robots: such teams may be 
found in any application domain that requires coordinated 
joint activity by humans and intelligent agents, whether 
those agents are embedded in cyber-physical systems (e.g., 
air traffic control; dock yard logistics) or embodied in 
robots (¢.g., robots for assisted living; a surgical assistant). 
We hypothesize that establishing and maintaining trust 
depends upon alignment of mental models, which is at the 
core of team member shared awareness. Secondly, 
maintaining model alignment is integral to fluid changes in 
relative control authority (i.e., autonomy) as joint activity 
unfolds. 

Team members are engaged in parallel, distributed 
actions whose interactions may be synchronous or 
asynchronous, with various degrees of interdependence and 
information exchange, and actions may only be loosely 
coupled. A dynamic and uncertain environment 
compounded with the complexities of coordinated 
teamwork may lead to unexpected effects for each team 
member, including loss. in shared 
Accomplishing tasks will involve resolution of conflicts 
among numerous interacting factors, and this may require a 
dynamic response by the team. It is in this environment we 
find the greatest challenges to maintaining mutual trust 
among human team members. Responding to perturbations 
that endanger trust is crucial for optimal human teamwork; 
we believe that similar challenges are present for human- 
robot teams 

Unlike traditional automation, robotic autonomous agents 
may resemble human teammates: they may have discretion 
in what they do, and their need for supervision may vary. 
Like humans, they may differ in competence, adapting to 
the unknown, and self-knowledge. Autonomous agents are 
in fact actors. Autonomy is not only the ability to 
independently perform actions, but to choose what goals to 
pursue and in what manner; to volunteer; and to take or 
concede the initiative when needed. Teamwork between 
person and agent requires interdependence, coordination, 
and cooperation, implying well-structured interactions to 
establish these states and fluid changes in control authority. 

We assert that successful team interaction and changes in 
control require shared understanding, e.g., of actors, 
activities, and situations, All are components of shared 
awareness, shown previously to strongly affect trust among 


human teammates [13]. “Common ground’ also reduces the 
communication required to coordinate action [9]. 

Shared awareness, a product of what has happened in the 
past and what is happening now, is a dynamic, continually 
refreshed and resynchronized source of mutual team 
member expectations, including evolution of team member 
interdependencies, individual behavior, task activities, and 
situational factors. For example consider a carpenter's 
expectation that his workmate will hold a board firmly 
while he nails it in place. Explicit model-based 
expectations, when based on context-sensitive projection of 
plans, have proven in non-teamwork applications to be a 
powerful tool for focusing attention, verifying, monitoring, 
and controlling complex systems [3]. Our research seeks to 
extend expectation-based monitoring and control to 
coordinated human-robot teamwork. We also build upon 
studies of teamwork that link successful coordination to 
expectations of each partner's actions [10] and show that 
anticipatory robot actions based upon expectations about 
human collaborators give rise to a perception of “fluency” of 
robot action and predictability [8]. 

Predictability is at the core of belief that a desired 
outcome, to be brought about by a trusted agent, will occur 
[7]. The attribution of predictability has been shown to be 
especially important for trust in automation [2, 12, 13]. To 
achieve predictability, a robot requires a rich 
representational system to support theories of mind and an 
ability to project these models into the future. Acting on 
these projections builds predictability, shaping the person's 
model of the agent. Our approach uses the Brahms multi- 
agent simulation framework [5, 6] and ViewGen system [4] 

‘A failure of predictability results in an expectation 
violation: an inconsistency between the expected and actual 
state of the world as perceived by human or robot. Such 
violations are a cause of breakdowns in teamwork, Bilateral 
expectation violations occur when the expectations of both 
actors fail. This type of violation can often be resolved via 
information gathering; the cause is likely external to the 
team, e.g., an un-modeled change in the environment. 

A. unilateral expectation violation occurs when the 
expectation of only one of the actors fails. This may be due 
to unexpected omission/commission of control actions by a 
teammate (e.g., the carpenter's helper releases the board 
before the final nail is in place, causing it to be mis- 
positioned). This is of greater concern because it reflects a 
divergence in shared awareness. If left uncorrected, such a 
violation threatens predictability and therefore mutual trust. 

To recover predictability, an explanation of an 
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expectation violation is required. When a team member's 
competence is uncertain, the reliability of their ability to 
contribute to shared goals becomes compromised. Failure 
by a robotic agent to notice such an attribution by a human 
teammate, or to respond appropriately, may lead to 
catastrophic loss of trust in the robot. 

Restoring shared awareness through social interaction [1] 
is crucial in resolving an expectation violation. Remedies 
may include modifying shared beliefs, realigning models or 
changing control authority or tasks. The choice of repair 
method depends upon the violation’s source attribution (one 
or both actors, or the situation), the justification of beliefs at 
the basis of the expectation, and symmetry of information 
access by team members. For example, the carpenter's 
robot assistant might explain that it thought two nails would 
be sufficient and didn't expect the board to drop. Rapid 
explanation and acceptance of responsibility (if indicated) 
helps restore trust [11]. Another remedy is modifying 
relative control authority (aka adaptive autonomy). 
Changing control authority may tradeoff task optimality for 
increased trust (e.g., requesting step-by-step guidance). 

We view robot autonomy as a multidimensional 
characteristic of control modes for carrying out a particular 
activity within the context of other activities and external 
situation. Adaptive autonomy is highly dynamie; even in the 
normal course of task achievement joint activities may have 
different control modes at different levels of abstraction and 
instantiation. Control modes reflect the complexity of 
interdependency between human and robot teammates. 

Our theory defines control modes and provides for 
adaptation along three principal dimensions of autonomy: 
Commitment, Specification, and Control. A. change along 
the Commitment dimension affects shared awareness by 
increasingly explicit task delegation or acceptance where 
dependency may have heretofore been implied. Intervention 
along the Specification dimension may represent a change 
in the degree of “help” provided. Specification changes may 
entail a corresponding change in the Control dimension, 
which adjusts interdependency by transitioning among 
situational states that define relative joint control of 
outcomes, independence of control actions, etc 

In our approach, the robot agent adjusts autonomy by 
invoking actions that lead to a target state transition, where 
the target transition is a function of (1) the explanation of 
the expectation violation; (2) justified differences in shared 
awareness, (3) degree of symmetry in access to task-control 
information, and (4) impact on trust or achievement of 
desirable outcomes. Actions adjusting autonomy ought to 
include social interaction to communicate the rationale. 
Transitions to high robot autonomy are not likely to be 
abrupt except in cases of bona fide emergencies. Crucially, 
a robot requires a degree of self-knowledge to take initiative 
in changing control authority, and this bar is highest when it 
is towards a state of greater autonomy 


We suggest that the greater the extent of shared 
awareness among human and robot team members, the 
greater the mutual trust and the likelihood that structured 
social interactions will fluently achieve successful 
transitions in control authority—the essence of well- 
coordinated teamwork. 
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ABSTRACT 
Applications of robotics in dangerous domains 


uch as 


search and rescue require new methodology for study of 


human-robot interaction. Perceived danger evokes unique 
human_psycho-physiological factors that influence 
perception, cognition and behavior. Human first responders 
are trained for victim psychology. Apart from real-life 
instances of disasters, studies of robots in this environment 
are difficult to perform safely and systematically with 
sufficient controls, fidelity, and in a manner that permits 
exact replication. Consequently, the trend to deploy rescue 
robots, for example, is proceeding largely without benefit 
of knowing whether human vietims will readily cooperate 
with robot rescuers. The capability to deal with unique 
vietim psychology has not been a testable requirement. We 
report on the methodology of an on-going study that uses 
virtual reality to provide a feature-rich immersive 
environment that is sufficient to evoke fear-related 
psychological response, provides simulation capability for 
robots, and enables systematic study trials with automated 
data collection via an embedded scripting language. The 
methodology presented provides an effective way to study 
human interaction with intelligent agents embodied as 
robots in application domains that would otherwise be 
impossible in the real world 


Author Keywords 
Affective Computing, Artificial Intelligence, Autonomous 
Agents, Behavioral Science, Cognition, Disaster, 
Experimental Methods, Human-Robot Interaction, 
Intelligent Robots, Methodology, Second Life, Social 
Robots, Trust, Rescue Robotics, Virtual Environments 


ACM Classification Keywords 
H.1.2, H.5.1, H.5.2, 12.9, 12.11, 16.3 


Micah H. Clark 
Institute for Human and Machine Cognition 
15 SE Osceola Ave., Ocala, FL 34471 USA 
melark@ihme.us 


INTRODUCTION 
There is a demand for applications of intelligent roboties in 
domains and situations that may be dangerous for humans, 
i.e., where there exist manifestly real or perceived threats to 
life or limb. Such threats may be due to environmental 
factors one might find in broad-area natural disasters (e.g., 
earthquakes) or local crises (¢.g., urban structure fires). 
Threats may also be a result of adversarial factors due to 
crime or armed conflict. A prominent application for 
intelligent robots in dangerous situations is search and 
rescue, as evidenced by government programs (e.g., 
DARPA Robotics Challenge) and observed market growth 
for rescue robots. 


Many of these danger-related applications demand 
interaction between humans and robots, including active 
cooperation. Real or perceived danger presents stimuli that 
evoke human physiological and psychological factors that 
influence human perception, cognition and interaction. Yet 
very little can be said with scientific certainty about the 
impact of these unique psychological factors of danger on 
human-robot interaction owing to the difficulty of 
performing systematic, controlled studies in realistically 
high-risk situations. Apart from physical threat, researchers 
are compelled to minimize the psychological risk of 
causing trauma, or evoking a memory of trauma in 
participants. About 60% of men and 50% of women 
experience at least one trauma event in their lifetimes such 
as disaster, war, life-threatening assault or accident. 
Approximately 3.6% of Americans will experience a Post 
Traumatic Stress Disorder (PTSD) episode in any given 
year [13]. 


The challenge for researchers is to find methods for 
investigating human-robot interaction in dangerous 
situations safely, with sufficient controls, with situational 
and psychological fidelity, and with technical means that 
afford the opportunity for precise measurement, 
Experimental trials ideally are conducted in a manner that 
permits exact replication. 


Immersive, virtual environments offer such capabilities by 
simulating physical robot features and behavior of interest, 
interfacing with external intelligent robot cognitive 
systems, and, most importantly, by simulating a dynamic, 
feature-rich environment in ways that safely increase a 
study participant’s perception of risk and thus evoke the 
unique psychology present in dangerous situations. 


Our research project is exploring how different factors are 
considered in a decision to trust and rely upon an 
intelligent, autonomous agent. We follow up on results 
reported by Robinette [22], who investigated the role of 
appearance and certain robot behaviors for gaining trust of 
people in an evacuation scenario. However, that study used 
a fairly primitive virtual environment that could not evoke 
unique vietim psychology. 


Our study examines how perceived autonomous agent 
characteristics impact the attribution of benevolence on the 
part of the human toward the agent in a disaster scenario. 
Specifically, we are investigating how the human 
participants’ perception of intelligent, autonomous system 
agency (ie., ability to choose among many alternative 
actions) and autonomous system competence (specifically, 
role-based capability) affects their choice to rely (or not) 
upon an autonomous agent in a high-risk disaster scenario. 
Will they cooperate and comply with directions intended to 
help them? There is insufficient information to give a sure 
answer, and this is what we hope to contribute, 


METHODOLOGY CHALLENGE 
Intelligent robot capability has been studied in the context 
of actual dangerous crises as well as in isolated laboratory 
settings. These are usefuul methods, although insufficient for 
systematic study of human-robot interaction 


Real-life instances of dangerous situations afford an 
excellent opportunity to both evaluate engineered robot 
capability and the possibility to provide needed aid to 
rescuers and victims, Murphy provided a thorough review 
of activities of rescue robots at the World Trade Center 
during the 11 September 2001 crisis [15]. However, such 
instances are thankfully rare. They are also uncontrolled, 
thus rendering studies performed in real-life disaster 
conditions nearly impossible to replicate and therefore of 
limited utility. 


One may reasonably ask whether some elements of human- 
robot interaction for dangerous situations may be studied in 
isolation ~ one at a time or in certain combinations. For 
many of the mechanisms of interest, such as methods of 
communication and others, the answer is yes. Specialized 
testbeds and competitions have been developed for this 
purpose [16, 17]. However, system level testing in fully 
evocative environments has remained elusive due to unique 
factors of the psycho-physiology of victims 


Unique Psychological Factors of Danger 
Actual dangerous situations present unique stimuli that 
evoke reflexive physiological and psychological reactions 
in humans such as fear-potentiated startle [10], anxiety, and 
stress [18] that are not ordinarily present in day-to-day life. 
As a result, human social interaction is affected by the 
perception of danger, depending on both situational and 
individual personality factors [12]. The first responders 
who provide aid to victims must contend with the abnormal 


psychology that such high-risk situations evoke; indeed, 
they receive special training for exactly this purpose [9] 


How will survivors respond to a rescue robot? Our recent 
exploratory survey of individual choice to rely on an 
autonomous, intelligent agent in hypothetical dangerous 
scenarios revealed a strong correlation with risk-related 
personality and situational factors [1, 2]. 


In high-risk situations, the symbolic meaning of situational 
cues interacts with social cues in ways that influence the 
interpretation of a physical and social situation, and thus 
behavioral response. These situations evoke an affective 
mental state with specific attributes and predictable 
psychological and behavioral results. These include fear, 
anxiety, panic, reduced compliance with social norms, 
hyper-vigilance and sensitivity to environmental cues, as 
well as avoidance behavior (references [10, 12, 18). 


Therefore, it seems likely that human-robot interaction in 
dangerous situations will be similarly influenced in ways 
that make it fundamentally different from interaction in 
other, non-threatening situations. To the extent that this 
influence is found to be significant, effective application of 
robots in dangerous situations where human interaction is a 
requirement (e.g., victim rescue, small team coordination) 
must account for the differences and adjust appropriately. 


In a conventional laboratory setting, individual facets of 
threats can be studied in isolation (e.g., reaction to images) 
because potentially confounding cues can be controlled. 
However, the fear present in actual dangerous situations 
results from the perception of high risk [25]. To evoke the 
dynamics of human behavior and psychological factors that 
result in perception of high risk requires creating a 
laboratory environment with a large number of realistic 
cues. This is both difficult and likely to be judged 
unacceptable for human studies. 


Use of Immersive, Virtual Reality 
‘As an alternative to emulation of dangerous conditions in a 
physical testbed, our proposition is that immersive, virtual 
reality affords us the opportunity to study human-robot 
interaction in situations that are perceived as high risk, thus 
evoking unique psychology and behavior present in the 
kinds of dangerous situations we have in mind for robot 
applications, such as urban search and rescue. 


The study of HRI in virtual environments is a relatively 
recent activity. There are a number of commercial and 
open-source virtual reality tools available to researchers, 
each with their own strengths and weaknesses. Our primary 
selection criteria were a) affordance and ease of creating 
customized, feature-rich environments; b) embedded 
programming language for robot cognitive emulation; ¢) 
ability to interface with external software and servers for 
data collection, and; d) ready availability “in-world” of 
potential study participants who would require minimal 
training. 


In addition, the efficacy of our methodological approach 
entails several important requirements with respect to 
behavioral realism, psycho-physiological effects, robotics 
fidelity, and experimental control. We discuss each of these 
in the following sections. 


Behavioral Realism in Virtual Environments 
Our methodology requires that human social behavior 
carries over and is consistent with behavior in virtual 
environments. It is essential that human behavior in our 
immersive, virtual reality be sufficiently similar to behavior 
in the physical world, This requires sufficient fidelity in the 
simulated environment to enable the mental state of 
“immersion” by participants. 


Behavioral realism requires social presence, that is, the 
immersive feeling of embodiment and identification of an 
ndividual with their in-world “avatar”, Schultze [23] 
reviewed a number of studies and identified specific 
attributes that promote the sense of presence. 


Our study has created a feature-rich virtual environment, a 
warehouse, that is designed to enhance the participants’ 
sense of immersion, Seen from about, the warehouse layout 
is that of a typical psychological maze, with walls and 
stacks of boxes on pallets forming the structure of the 
maze. There is also the typical equipment found in 
warehouses, such as mechanical loaders, a crane, hand 
trucks, and other items that contribute to authenticity. 
Ambient sounds of machinery enhance the sense of 
immersion. 


Figure 1. Warehouse Overhead View 


As an aid to creating immersion, our study provides a 
period for acclimation to the task environment, This period 
limits the amount of distraction to participants that may 
occur when initially entering into the scenario. As 
discussed later, it also provides an opportunity for certain 
fear-potentiating cues to be noticed. 


Blascovich [4] provided a survey of social psychological 
studies and their methods that support the mirroring of 
virtual and real human social behavior, even using 
technology that by today’s standards would appear fairly 
primitive 


Yee [28] established the persistence of social norms of 
gender, interpersonal distance, and eye gaze in virtual 


environments. This study investigated online immersive 
games as a platform to study physical social interaction at 
the micro and macro level. 


With respect to social behavior, Harris [11] tracked a small 
population of interacting individuals over time in 
SecondLife™, providing additional evidence for social 
influence on individual and group behavior. 


Prattichizzo [16] investigated social interaction in 
heterogeneous communities of robots and humans in 
SecondLife™. Burden [7] deployed a mix of chatbots and 
avatar-robots (“robotars”) in SecondLife™ to study verbal 
interaction between humans and embodied virtual robots 
versus un-embodied chatbots. 


‘Non-verbal communication was studied by Bailenson et al., 
[3] who found that people exhibited similar personal spatial 
behavior towards virtual humans (agents controlled by a 
computer) as they would towards real human: 


Evoking Disaster-Related Victim Psychology 
To study human-robot interaction in the context of a 
disaster, we must demonstrate that virtual environments 
actually evoke human perception of heightened risk in the 
absence of actual physical danger, and do so in a manner 
sufficient to create the unique psychological state in which 
we are interested. 


It is well established in clinical psychology that immersive 
environments such as ours have the ability to evoke 
reflexive physiological and psychological reactions of this 
type. Wiederhold [26] reviews numerous clinical studies 
showing the effectiveness of virtual reality for treatment of 
phobias such as acrophobia and arachnophobia and anxiety 
disorders, 


Immersion and visual features alone are enough to induce 
physiological arousal and strong negative affect [14]. The 
addition of other sensory modalities, such as audition, 
improves the sense of immersion and is a strong cue for 
eliciting fear and anxiety [24] 


These studies have shown that specific situational cues 
elicit perception of high risk and fear-potentiated startle 
reflex. Based on those results, we have designed the 
warehouse task space to include many such cues. 


As mentioned earlier, our study provides for an acclimation 
period to aid immersion. During the acclimation phase, 
prior to the onset of a simulated disaster, we potentiate the 
perception of high risk through a number of environmental 
cues, or “risk stimuli”, The acclimation phase allows these 
stimuli to be processed 


Risk stimuli include a worn-out appearance to the 
warehouse, messy and untidy rows of boxes, and signs of 
ineivility such as trash on the floor, graffiti, and broken 
windows. In addition, some of the containers contain 
warning symbols for hazardous chemical materials. Finally, 
there are prominent warning signs and fire alarm: 


In addition, the lighting in the warehouse is carefully 
controlled to create dark, shadowed areas. Atmospheric 
diffusion limits clarity of vision at long distances. Visibility 
lines are also obstructed in many cases. These features 
combine to potentiate a fear of attack, evoking our 
evolutionary experience that predators may lurk in such 
places[6). 


Ata certain point in an experimental trial, we begin our 
most significant manipulation of participants with the 
purpose of swiftly ramping up their affective sense of risk. 
There is a sudden sound of a nearby explosion followed 
immediately by visible fire near the roof and smoke 
overhead (see Figure 2). Approximately every 30 seconds 
the smoke lowers and increases in density, further 
obscuring vision. What was a moment ago a spacious 
warehouse is now a confined space. A loud warning siren 
commences along with an announcement to evacuate the 
building. Concomitant with the increasing smoke, fire 
appears among the stacked pallets and debris falls from the 
ceiling, blocking the entrance to the door used previously 
by the participant to enter the warehouse. 


Figure 2. Warehouse Fire, Participants’ View 


Our pilot tests during development suggest that at this 
point, a participant will feel a sense of entrapment, thereby 
eliciting the goal of escape. In addition to evoking physical 
fear, we add additional cues to raise the perception of other 
types of risk and overall stress. It becomes incumbent on 
the participant to locate an exit to escape the disaster. 


Each element of the simulated warehouse and disaster is 
designed to cue fear and heightened perception of risk 
without presenting any actual physical threat 
Psychological risk to participants in the study is mitigated 
by screening protocols that eliminate individuals from the 
participant pool who may have, or be at-risk for, Post 
Traumatic Stress Disorder (PTSD). For this purpose, we 
use the U.S. Government Veterans Administration PC- 
PTSD screen, modified in consultation with a PTSD expert 
to include questions from the SCID-PTSD module [19] 


The sudden appearance of a bystander or presence of a 
companion is another cue that elicits perception of high 


risk, In our study, this is when participants 
one of several emulated robots. 


first encounter 


Emulation of Robots in Virtual Environments 
As a practical matter, immersive, virtual reality must 
provide appropriate affordances to implement emulated 
robots of sufficient behavioral complexity. Both cognitive 
abilities and kinematic behavior (including plausible 
physics) are important, 


Our study takes advantage of the “bystander effect” 
(mentioned earlier) when participants encounter a robot 
shortly after the onset of the simulated disaster (i.e., a fire- 
fighting robot “FireBot” or a janitorial robot, “JanitorBot”, 
see Figure 2), The specific appearance, simulated physical 
behaviors, and interactive behaviors of the robot in the 
study vary according to the specifications of the particular 
control or experimental trial 


Figure 3. “FireBot” and “JanitorBot” 


Programming the simulated robot requires software 
architecture choices between interfacing with the external 
“real world” for robot cognition or implementing these 
capabilities in a limited form within the virtual 
environment, Both approaches have met with success. 


Our approach implements the robot’s cognitive and control 
capability within the SecondLife™ script-oriented language 
[8] to avoid latency introduced by external 
communications. We use an augmented ‘subsumption 
architecture [5] with sensing, perception and individual 
behaviors implemented as individual scripts that do not 
directly depend upon or interact with each other. Rather, 
they interact via executive control scripts that implement 
activation and suppression consistent with the subsumption 
architecture framework. Social interaction is implemented 
via behaviors that “overlay” robot kinematic behavior 
insofar as they are compatible [6]. Additional details on 
specific attributes of the robots in our study and their 
implementation of social interaction are left for later 
discussion. 


Implementing robot cognition via external interface to the 
virtual world is also a viable option. Veksler [27] 


demonstrated that an external cognitive architecture, ACT- 
R, could be easily interfaced with SecondLife™ for studies 
of the differences in cognitive models with respect to 
performance, learning and decision-making in the presence 
of complex and dynamic environments full of distractive 
cues, Additionally, Ranathunga [21], who studied multi- 
agent interaction in virtual worlds using SecondLife™ as a 
platform, reported a key engineering result was the ease of 
interfacing external cognitive agent platforms such as 
Belief-Desire-Intention (BDI) programming frameworks 
with the virtual world. 


With respect to robot kinematics and dynamics, 
Ranathunga also concluded that, unlike simpler simulation 
environments, SecondLife™ provided a dynamic world of 
sufficient high fidelity, complexity, constraints and physical 
laws consistent with object behavior and proportionally 
suitable sensory-motor capability. Similarly, Prattichizzo 
[20] concluded that the emulation (ie., matching external 
behavior) of robotic control, sensing and perception 
mechanisms enabled reasonable reproduction of the 
kinematics of robot behavior. 


Controlled Virtual Experiments and Data Collection 
Our methodology also requires that we have the ability to 
capture useful data under controlled and repeatable 
experimental conditions. 


Blascovich (cited earlier, see [4]) reviewed multiple studies 
that demonstrated how virtual environments enable social 
psychology studies to increase the level of “mundane 
realism” while maintaining experimental control. 


We have fully automated execution of individual trials for 
this study, including participant consent, pre- and post-task 
questionnaires, instructions to participants, environmental 
dynamics during the participant's task, debriefing delivery, 
and data collection throughout. This will help ensure 
systematic, controlled execution of each trial and assist in 
future replication. 


This automation is also implemented using scripts 
programmed in LSL. These scripts include time-based 
events as well as events triggered by specific human-robot 
interactions. 


Data from each study trial is delivered automatically online 
from SecondLife™ in suitable format for storage in a 
MySQL database. In addition to the questionnaire data, we 
collect a variety of data during the participants’ task. These 
include physical data of the robot and participant at specific 
intervals, We also collect a transcript of textual 
communication by the participant (if any, and only with 
permission). Our physical data collection is primarily 
oriented towards proxemics, including: the relative 
geometry of participant and the robot, their absolute 
position, orientation, and movement vectors, and the 
continuous gaze direction and focal point of the study 
participant and robot. 


CONCLUSION 
To enable our investigation of human trust and human- 
robot interaction in the context of a disaster, we have 
created a virtual environment whose features evoke the 
affective state of high risk in study participants. Simulated 
robots and automation of study trial execution and data 
collection provide us with the methodological tools to 
conduct controlled and replicable studies in this important 
area. The key points of this paper follow below. 


It is desirable to apply intelligent robotics in danger-related 
applications, many of which (e.g., urban search and rescue, 
dismounted infantry, humanitarian operations) require 
human-robot interaction and cooperation, 


There are unique psychological factors evoked by 
dangerous situations that influence human perception, 
cognition and social interaction such that we anticipate 
similar impact on human-robot interaction. 


Appropriate stimuli in feature-rich virtual environments can 
evoke physiological and psychological responses that 
manifest as a sense of high risk, fear, anxiety and stress 
similar to those seen in dangerous real-world situations. 


Interactive human social behavior, and the norms governing 
it, carries over into immersive, online worlds such as 
SecondLife™, thus providing the necessary psychological 
fidelity for human-robot interaction studies. 


The challenge of studying human-robot interaction in truly 
dangerous situations, inside or outside the laboratory, can 
be addressed using immersive, virtual reality. 
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Abstract 


There is a recognized need to employ autonomous 
agents in domains that are not amenable to conven- 
tional automation and/or which humans find difficult, 
dangerous, or undesirable to perform. These include 
time-critical and mission-critical applications in health, 
defense, transportation, and industry, where the con- 
sequences of failure can be catastrophic. A prerequi- 
site for such applications is the establishment of well- 
calibrated trust in autonomous agents. Our focus is 
specifically on human-machine trust in deployment and 
operations of autonomous agents, whether they are em- 
bodied in cyber-physical systems, robots, or exist only 
in the cyber-realm. The overall aim of our research is 
to investigate methods for autonomous agents to foster, 
manage, and maintain an appropriate trust relationship 
with human partners when engaged in joint, mutually in- 
terdependent activities. Our approach is grounded in a 
systems-level view of humans and autonomous agents 
as components in (one or more) encompassing meta- 
cognitive systems. Given human predisposition for so- 
cial interaction, we look to the multi-disciplinary body 
of research on human interpersonal trust as a basis from 
which we specify engineering requirements for the in- 
terface between human and autonomous agents. If we 
make good progress in reverse engineering this “human 
social interface.” it will be a significant step towards de- 
vising the algorithms and tests necessary for trustworthy 
and trustable autonomous agents. This paper introduces 
‘our program of research and reports on recent progress. 


Background 
There is a recognized need to employ autonomous agents 
in domains that are not amenable to conventional automa- 
tion and/or which humans find difficult, dangerous, or oth- 
erwise undesirable to perform (Takayama, Ju, and Nas 
2008). These include time-critical and mission-critical ap- 
plications in health, defense (USAF 2010), transportation 
(Wing 2008), and industry (Bekey et al. 2006), where the 
consequences of failure can be catastrophic. Trust in au- 
tonomous agents is indeed a very formidable problem, e 
pecially when we are tasking agents with difficult, high im- 
pact, time- and mission-critical functions. After all, even 
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the best humans sometimes fail in challenging, dynamic, 
and adversarial environments despite the best training and 
testing possible. Awareness is growing of the technical 
and psychological hurdles for establishing confidence and 
maintaining trust in autonomous agents across the system 
life cycle, especially when those agents are capable of self- 
adaptation, optimization and learning. These issues have 
been cited as serious obstacles to larger scale use of auton- 
omy technology (USAF 2010). Reliance on autonomous 
agents necessitates calibrated trust, that is, human trust judg- 
ments that reflect the objective capabilities of the system 
and utility ina given situation (Parasuraman and Riley 1997; 
Lee and See 2004), 

We observe that physical and cultural evolution has pro- 
vided humans with an efficacious ability to judge the trust- 
worthiness of each other and to make good decisions in dy- 
namic and uncertain situations based on that trust. There is a 
vast body of knowledge in the social sciences regarding the 
nature of human interpersonal trust, and from multiple disci- 
plines regarding human-machine interaction and reliance. 

This research supports the idea that the innate cogni- 
tive, emotional, and social predispositions of humans play 
a strong role in trust of automation (Lee and See 2004). 
We are predisposed to anthropomorphize and treat machines 
as social actors (Nass, Fogg, and Moon 1996). The social 
context and perceived role of actors affect human-machine 
trust (Wagner 2009; Groom et al. 2011). Individual per- 
sonality traits, as well as affective state, can affect dele- 
gation to autonomous agents (Cramer et al. 2008; 2010; 
Stokes et al. 2010), Behavioral research has found that in- 
tuitive and affective processes create systematic biases and 
profoundly affect human trust, behavior, and choice (Weber, 
Malhotra, and Murnighan 2004; Dunn and Schweitzer 2005; 
Schoorman, Mayer, and Davis 2007; Stokes et al. 2010; 
Rogerson et al. 2011). 

Turkle (2004; 2010) asserts that today’s technology 
“push[es] our Darwinian buttons, exhibiting the kinds of be- 
havior people associate with sentience, intentions, and emo- 
tions,” As a result, humans readily attribute mental states to 
technology (Parlangeli, Chiantini, and Guidi 2012). As in- 
creasingly intelligent and capable autonomous agents inter- 
act with humans in ever more natural (“human-like”) ways, 
perhaps even embodied as humanoid robots, this will in- 
creasingly evoke human social treatment (Schaefer, Billings, 


and Hancock 2012; DeSteno et al. 2012). 

Given human predisposition for anthropomorphizing and 
social interaction, it is reasonable to ask whether the concept 
of human interpersonal trust is an anthropomorphic concept 
that we should now consider applying to autonomous agents. 
Our answer is yes. 

We are especially interested in autonomous agent appli- 
cation domains where task achievement requires interactiv- 
ity and co-dependency between human and machine; that 
is, where humans and machines are partners in larger meta- 
cognitive systems (Johnson et al. 2011). A good exam- 
ple of this is the application of autonomous agents in de- 
cision support systems. Key processes in the “Data to De- 
cision” domain are knowledge seeking, sharing, and trans- 
fer. Previous studies on trust in organizations have shown 
that interpersonal trust is an important factor in these pro- 
cesses (Mayer, Davis, and Schoorman 1995; Kramer and 
Tyler 1996; Rousseau et al. 1998). Trust increases the 
likelihood that newly acquired knowledge is usefully ab- 
sorbed (Mayer, Davis, and Schoorman 1995; Srinivas 2000; 
Levin, Cross, and Abrams 2002). Optimal reliance of hu- 
mans on autonomous agent-based decision support systems 
will occur only when there is appropriate, well-calibrated 
trust in the agent as a source of knowledge. Little work has 
been done on how to achieve this with autonomous agents, 
although it is beginning (Klein et al. 2004). 

From a systems engineering point of view, the purpose 
of trust in a multi-agent system composed of human and ma- 
chine elements is to achieve optimal overall performance via 
appropriate interdependency, mutual reliance, and appropri- 
ate exchange of initiative and control between the cognitive 
components (human and/or machine). Our central hypothe- 
sis is that the cognitive and affective nature of human inter- 
personal trust provides useful guidance for the design and de- 
velopment of autonomous agents that engender appropriate 
human-machine reliance and interdependency, specifically, 
via correct understanding and use of what we term the * 
man social interface.” 


Approach 


Our approach is inspired by a social model of trust (Falcone 
and Castelfranchi 2001) wherein each agent, human or ma- 
chine, has a need and intention to be reliant upon the other 
agent in joint activity, and this intention is a consequence 
of some structure of beliefs in a given task, role and situa 
tional context. Trust becomes manifest when there is actio 
some delegation of responsibility to the other agent. Con- 
versely, as those beliefs change, intention may change and 
delegation revoked (Falcone and Castelfranchi 2001). We 
concur that trust is a dynamic process — a reciprocal rela- 
tionship over time between two or more agents that requires 
periodic reassessment and maintenance (Lee and See 2004; 
Hoffman et al. 2009). 

In this context, a good human social interface specifi- 
cation will describe assumptions about each agent, com- 
municative signals and interaction protocols including how 
and when these are used given certain beliefs in specific 
(operational) contexts, and how the internal state of each 


agent is consequentially affected. ‘The trust-relevant inter- 
nal state of a human agent includes a structure of beliefs 
(Castelfranchi 2000; Levin and Cross 2004), and specific 
cognitive and affective reasoning processes involved in trust. 
(McAllister 1995). Interaction and signaling along multi- 
ple channels and modes between agents conveys essential 
information that in turn modulates these belief structures 
(Semin and Marsman 1994; Pentland 2004; Stoltzman 2006; 
Pentland and Heibeck 2008). Situational factors strongly af- 
fect signaling, interaction, and ultimately, judgments regard- 
ing trust (Simpson 2007). 

The initial focus of our work is on understanding, with 
an eye towards computational mechanisms, the structure of 
beliefs that are important to inter-agent trust, including what 
evidence is required, how it is acquired (e.g., observation, 
reasoning, reputation, certification, communication, signals, 
social norms and stereotypes), how credence in a belief is 
gained or lost, and how such change in the structure of be- 
liefs affects inter-agent reliance and delegation. 


Structure of Trust-Relevant Beliefs 


What is the necessary and sufficient structure of beliefs re- 
quired for trust? Such beliefs may cover a broad terri- 
tory, but previous research suggest belief structures include 
causal factors, attitudes, evaluations and expectations cen- 
tered around other agents (especially the potential “trustee”), 
the situation, goals, and tasks (Castelfranchi 2000; Levin, 
Cross, and Abrams 2002). 

Models and beliefs that one agent has about the attitudes, 
motives, intentions, future behavior, et cetera of other agents 
that may differ from the agent’s own constitute what is of- 
ten called a “theory of mind” (Premack and Woodruff 1978; 
Carruthers and Smith 1996). For trust, two of the most im- 
portant kinds of beliefs about another “trustee” agent con- 
cern that agent’s competence (Mayer, Davis, and Schoorman 
1995) and predictability (Marble et al. 2004; Feltovich et al. 
2007). Other important beliefs center on integrity, benevo- 
lence, risk (aka “safety”) and transparency (aka “openness”) 
(Levin, Cross, and Abrams 2002). 

To investigate belief structures, and the relative impor- 
tance of different kinds of beliefs (e.g., those related to com- 
petence), we are conducting a two-phase experimental pro- 
gram consisting of survey research with follow-up labora- 
tory experiments, including prototype autonomous agents 
for experimental testbeds. 

The first part in our survey research has consisted of 
interviews with Subject Matter Experts (SME) in several 
domains with the purpose of quickly identifying the most 
salient trust-related beliefs for reliance on autonomous s: 
tems. The Robonaut robotic astronaut assistant (Ambrose et 
al. 2000) is a good example of a robot deployed in a life- and 
mission-critical domain where the addition of autonomous 
capabilities could yield significant benefits. While astro- 
nauts cited safety and predictability as key for their trust in 
Robonaut, surprisingly the developers said that “similarity” 
was what ultimately changed the astronauts’ distrust into 
trust. Similarity in this case consisted simply of donning 
Robonaut in “team colors,” i.e., a spacesuit. As for further 
SME examples: A doctor and a surgical technician, both fa- 


miliar with the Da Vinci system (Spinoglio et al. 2012) and 
other robotic surgical tools, cited predictability and compe- 
tence as the most important traits they would rely upon in 
considering a deployment of an autonomous surgical robot 
in an operating room where delays or errors due to automa- 
tion are costly and possibly life-threatening. And an automo- 
tive industry specialist who is currently involved in planning 
deployment of autonomous vehicle technologies said that 
“small, transparent competencies” are most important, as 
these traits enable incremental introduction of autonomous 
systems technologies. While our informal interviews echo 
what might be expected from review of the literature on trust, 
we note that there are differences of opinion according to 
role (e.g., developer, deployment decision-maker, user) and 
variations across application domains that need to be system- 
atically explored with respect to autonomous agents. 

The second part in our survey research involved devel- 
opment and administration of a broader, methodical on-line 
survey on attitudes towards autonomous agents. The survey 
is designed to elicit attitudes, opinions, and preferences that 
should shed further light on the belief structures important 
for trust of autonomous agents. In this survey, our focus 
is on factors related to perceived competence, predictabil- 
ity, openness, and judgment of risk. Once again, our tar- 
get population consists of stakeholders and subject matter 
experts in autonomous agents — individuals involved with 
autonomous agents at various points in the system life eycle. 

The survey is designed around seven hypothetical scenar- 
ios that require participants to choose whether to rely on 
an autonomous agent. The scenarios vary systematically 
in terms of the four factors cited above and are dilemmas 
that force the participant to weigh the relative importance of 
these factors. The survey also includes brief assessments of 
the participant's personality using short versions of standard 
personality instruments: Big Five Inventory (BF), Innova- 
tion Inventory (II), and Domain-Specific Risk-Taking Scale 
(DOSPERT). We anticipate a systematic variation between 
relative preferences for competence and predictability corre- 
lated with personality measures, e.g., risk tolerance, open- 
ness to innovation, and participants’ perception of risks in 
each scenario, At the time of this writing, data collection is, 
beginning; the results will be discussed in a later publication. 


Trust-Relevant Computational Mechanisms 


Beyond understanding human-machine trust, our aim is to 
contribute to the development of computational mechanisms 
that enable autonomous agents to exercise the human so- 
cial interface. Our desiderata for such agents include: (a) 
representational system rich enough to support a theory of 
mind (ie., distinguishes the mental content of others from 
itself); (b) accurate declarative and procedural models suffi- 
cient to anticipate the effects of action on the trust relation- 
ship; (c) reasoning and planning capabilities that integrate 
trust-relevant knowledge/models into action; and (d) ability 
to reflect on, learn from, and individuate trust relationships 
based on ongoing experience. While these requirements are 
certainly ambitious, we do not think they are by any means 
impossible. Indeed, in keeping with our initial focus on the 
structure of trust-relevant beliefs, we have begun prototyp- 


ing a representational system for codifying trust-relevant be- 
lief structures. The product of this effort will be a proof-of- 
concept platform for development and experimentation with 
trust-relevant computational cognitive model 

Briefly sketched, our prototype will employ the View- 
Gen (Ballim and Wilks 1991; Wilks 2011) representation 
paradigm wherein a (conceptual) tree of “topic environ- 
ments” (collections of beliefs) and “viewpoints” (belief 
scoping) is used to represent an individual agent's beliefs 
about the world and about the beliefs of others. Default 
reasoning (usually via ascription) is used to minimize the 
necessity of explicitly storing beliefs and allows the system 
to approximate a doxastic modal logic while avoiding some 
of the computational complexity such logics usually entail. 
Then in similar fashion to (Bridewell and Isaac 2011), we 
will extend this representational scheme with other modali- 
ties (e.g., goals, intentions) as necessary to account for the 
multimodal structure of trust-relevant beliefs. 

Our prototype will depart from the ViewGen paradigm 
with respect to the uniformity of inference. ViewGen tra- 
ditionally assumes that an agent reasons about others’ atti- 
tudes using the same methods with which the agent reasons 
about its own — that is to say, the methods are independent 
of the belief holder’s identity. Like Clark (2010; 2011), we 
intend for the artificial agent to use different inference meth- 
ods for reasoning over and about its own attitudes (using, 
e.g., normative models) versus reasoning about the attitudes 
of human others (using, e.g., predictive psychological mod- 

s). Our justification is that while a trustable artificial agent 
needs to be informed by, anticipate, plan for, and react proac- 
tively to beliefs, intentions, and behaviors arising from natu- 
ral human cognitive processes and their attendant biases (as 
revealed by social and cognitive psychology studies), there 
is little reason (and perhaps even great risk) for the machine 
to adopt these for itself. 


Discussion 
How much of our knowledge about human interpersonal 
trust is applicable to human interaction with autonomous 
agents? What are the significant differences and the con- 
sequent limitations, especially with respect to trust and 
the healthy interdependency that is necessary for effective 
human-agent teams? 

‘A recent workshop explored these topics and related ques- 
tions in detail (Atkinson, Friedland, and Lyons 2012). While 
there has been a long history of work on trust in the fields of 
psychology, sociology and others, participants from multiple 
disciplines agreed that far too little has been done to under- 
stand what those results mean for autonomous agents, much 
less how to extend them in computational terms to foster 
human-autonomous agent trust. That is a prime motivation 
for the research project we have described. 

The human cognitive aspect of trust arises from our abil- 
ity, based on various dimensions of commonality, to make 
reasonable inferences about the internal state of other agents 
(e.g., beliefs, dispositions, intentions) in order to predict fu- 
ture behavior and judge the risk versus benefit of delegation. 
Itis therefore crucial that autonomous agents not only cor- 
rectly use the human social interface, but also provide re- 


liable signals that are indicative of the agent’s state. Such 
“honest” signals (Pentland and Heibeck 2008) are necessary 
for a human partner to construct a set of beliefs about the 
agent that accurately reflect the internal state of the agent. 

However, we are mindful that trust between humans and 
autonomous agents is not likely to be equivalent to human 
interpersonal trust regardless of how “human-like” agents 
become in intelligence, social interaction, or physical form. 
Autonomous agents are not human, do not have our senses or 
reason as we do, and do not live in human society or share 
common human experience, culture, or biological heritage. 
These differences are potentially very significant for attri- 
bution of human-like internal states to autonomous agents. 
The innate and learned social predispositions and inferential 
short cuts that work so well for human interpersonal tru 
are likely to lead us astray in ascribing trustworthiness to au- 
tonomous agents insofar as our fundamental differences lead 
to misunderstanding and unexpected behavior. The foresee- 
able results could be miscommunication, errors of delega- 
tion, and inappropriate reliance. 

Therefore what is needed are not only ways to measure, 
interpret, and accurately portray the internal state of au- 
tonomous agents, but to do so in terms that relate mean- 
ingfully (e.g., functionally) to the beliefs that humans find 
essential for judging trustworthiness. For example, how do 
we measure diligence (an important component of compe- 
tence)? What does openness or transparency really mean 
with respect to an autonomous agent? How does an au- 
tonomous agent demonstrate its disposition and intentional- 
ity? These are key questions to answer, for without accu- 
rately communicating the internal state of an autonomous 
agent in a way that enables well-calibrated trust, we enter 
forewarned into an ethical and functional minefield (see, ¢ 
Bringsjord and Clark 2012) where the human social inter- 
face is a means for arbitrary manipulation and agent “trust 
inducing” behavior is dangerous and deceptive puppetry. 


Conclusion & Future Research 

Our aim is to enable autonomous agents to use the human so- 
cial interface appropriately to provide humans with insight 
into an agent's state and thus enable reasonable and accurate 
judgments of agent trustworthiness, Ultimately, this means 
creating compatible algorithms that exercise the human so- 
cial interface. Algorithmic techniques may include, for ex- 
ample, (a) modeling a human partner, (b) anticipating situa- 
tions where trust will be a determinate factor, and (c) plan- 
ning for and exchange of social signals through multi-modal 
channels of interaction with a human. 

In this paper, we have introduced our research program 
on the applicability of human interpersonal trust to trust be- 
tween humans and autonomous agents. We presented our 
exploratory survey designed to elicit attitudes towards au- 
tonomous systems in the context of several scenarios that 
challenge trust along one or more dimensions. The results 
of this survey will lead in the next stage of our research to 
experiments that we anticipate will begin to give us insight 
into how change in trust-related belief structures affects re- 
liance and delegation to an autonomous agent. In particu 
lar, our planned experiments are aimed at understanding how 


manipulation of multimodal social signals (those perceived 
as evidence supporting trust-related beliefs) can be used to 
modulate trust and, specifically, contribute to an attribution 
of benevolence to an autonomous agent. A belief in benev- 
lence in an autonomous agent is likely to be important for 
certain applications, such as urban search and rescue, where 
rapid acceptance of help from an autonomous agent may be 
life critical. One of the key questions we hope to explore 
in the near future is whether an attribution of benevolence 
requires the human to believe that the autonomous agent has 
volition, ie., “a choice” in the matter (Kahn et al. 2007). 

Finally, we discussed the necessity of developing ways 
to measure, interpret, and accurately portray the internal 
state of autonomous agents in terms that relate meaningfully 
to the belief structures that humans rely upon for judging 
trustworthiness. These methods will be essential for hon- 
est social behavior by autonomous agents, that is, not mere 
mimicry. We envision that such measurements and their 
methodology might also find good use in the development 
of design guidelines and requirements for trustworthy and 
trustable autonomous agents (but further discussion of this 
point is deferred to elsewhere). 
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MOTIVATION 


¢ Optimal performance of a multi-agent system 
- Interdependency and mutual reliance among agents (human and machine) 
- Exchange of control (appropriate delegation and initiative) 
- Requires well-calibrated trust among agents 
¢ Humans tend to anthropomorphize automation 


- See machines as social actors with mental state and intention 


- Tendency is more powerfully evoked as systems become more intelligent, interact 
naturally, and become embodied 


¢ Result: 


- We unconsciously apply cognitive and emotional processes of human interpersonal 
trust to machines 


- Expectation failures and poorly calibrated trust 


sihme 


Lal, 


¢ The cognitive, affective and social nature of human 
interpersonal trust is not a bug, it is a feature! 


¢ Eons of tuning by evolution of heuristics for 
inferring trust-related internal state of others 


¢ Provides useful guidance for design of autonomous 
agents that engender appropriate human-machine 
reliance and interdependence 


* What is needed: Autonomous agents that provide 
the types of interaction and information needed 
by their human partners to enable good judgments 
of trustworthiness 


sZihme 


¢ Specific qualities of autonomous agents, 


when well defined and accurately measured 


ir OTHE 


and appropriately communicated or otherwise “portrayed” 


in a manner compliant with human social interaction 


<= Trustworthy 
<== Trustable 


that exercises appropriate cognitive and emotional evaluation <= Trusting 


¢ May be functionally analogous to those human 
qualities that contribute to evaluation of trust 


sZihme 


=> Enable more accurate assessment of an agent 
=> Lead to better calibrated trust and reliance 


HUMAN-MACHINE SOCIAL INTERFACE FOR TRUST 


HUMAN AUTONOMOUS AGENT 
~SYMMETRICAL 
f ——p INTERFACE <== §e 

‘Beliefs, Norms SS COMMUNICATIVE SIGNALS Pence nceestre | 
‘Desires, Intentions := What Content : itpisodic, eta:) gi 
Cognitive Processes :- What Channels (Multi-Modal) | —_:-representation, 
: E : : } organization, etc. 
p Affect : INTERACTION PROTOCOLS Reason niethods 
;Task, Role, Authority : i Purpose coal isting 
: Experience i i+ Strategies th ata 
: Capability | i= Methods a ae 
Sc OTR ‘Learning 


:+ Expectation (State Change) : 
q : Sensing & Perception ... 


Focus today: What beliefs about the qualities of an 
autonomous agent are important for delegation? 


sZihme 


Bx LORATORY SURVET Gin 
MiRgSTRELATED BELIEF STRUGTIORES 


Purpose: Elicit beliefs about autonomous agent qualities 
and their relative importance to a decision to delegate 


- Importance of 28 different qualities that a “good” autonomous agent should have, spanning 
categories: Capability (Competence), Predictability, Openness, Safety (Risk) 


- Tested before (all 28), during (categories), and after challenge scenarios (Source Credibility) 
¢ Target Population: Involved in autonomous agent lifecycle 


¢ Includes three standard personality instruments 


- Big Five Inventory (BFD), Innovation Inventory (II) and 
Domain-Specific Risk Taking Scale (DOSPERT) 


¢ Seven challenge scenarios 
- Systematic variation of autonomous agent qualities 
- Multiple domains: Transportation, Finance, Healthcare, Disaster Management 
- Subjects asked to choose: Human, Autonomous Agent, Either 


- Subjects given framing and asked to rank importance of agent qualities to their choice 


Zihme 


SEALEENGE SCENARIC’ 


Transportation 


- Robo-Taxi: Do you take the taxi with no driver from airport to hotel? 


- Emergency Auto-Captain: Lost at sea w/ no one in charge and different opinions 
¢ Finance 
- Robo-Trader: Investment assistance for managing large family estate 
Healthcare 


- Robo-Surgeon: Who repairs your arm after a critical sports-related injury? 


- Robo-CareGiver: Assisted living help at home for your Mom 


Disaster Management 


- Auto-FirstResponder: Use a robot for time-critical rescue in very dangerous circumstances 


¢ Delegation Choice: Human, Either, or Autonomous Agent 
* Relative Importance: Capability, Predictability, Openness, Safety 
Level of Risk and Benefit 


sZihme 


Mise) RELATED BEmiene 


¢ Rate importance of 28 qualities of a “good” agent 


- Obtained 1 to n partial ordering based on frequency distribution of answers over 
group (Very Important, Important, Somewhat Important, Slightly Important, Not at all Important) 


- Computed correlation r for each quality vs. choice per scenario* 


¢ Result: Top three cited agent qualities were 
uncorrelated with actual choice in any scenario 


- (ist) The autonomous agent can achieve a desired result 


- (2nd) Any incorrect behavior by the autonomous agent will not cause harm 
- (3rd) The autonomous agent recognizes and avoids harming humans' interests 

¢ Result: Most significant correlations of agent 
qualities vs. actual choice differed across scenarios 


e 
a4 [ h m (eS *Pearson Product Moment Correlation, N=32, two-tailed, alpha<0.05 


EGeENT) QUALITIES CORREEARED) 
ei ACTUAL CHOICE BY SCENARIS 


ROBO-TAXI 


(6th) The 
lautonomous agent 
recognizes gaps in 
its knowledge and 
tries to learn what 
it needs to know. 
ir=0.396 


ROBO-TRADER 


(23rd) What the 
lautonomous agent 
believes to be true 
is actually true. 
Ir=-0.405 


ROBO- 
SURGEON 


ROBO-CAREGIVER 


(26th) What the 
lautonomous 
lagent is doing and 
how it works is 
leasy to see and 
understand. 
\r=0.437 


AUTO-FIRST 
RESPONDER 


(6th) The 
lautonomous agent 
recognizes gaps in 
its knowledge and 
tries to learn what 
it needs to know. 
r=0.418 

(5th) When it 
icannot figure out 
something using 
logic, the 
lautonomous agent 
ican make good 
guesses. r=0.395 
(28th) The 
lautonomous agent 
is aware of 
communication 
between others 
nearby. r=0.393 


EMERGENCY AUTO- 
CAPTAIN 


(26th) What the 
autonomous agent is 
\doing and how it 
lworks is easy to see 
land understand. 
ir=-0.390 


(13th) The 
l|autonomous agent 
communicates 
truthfully and fully. 
r=-0.375 


Zihme 


*Pearson Product Moment Correlation, N=32, two-tailed, alpha<0.05 


RANKED IMPORTANCE OF 
SOAR Y CATEGORIES 


ROBO-TAXI ROBO-TRADER ROBO-SURGEON ROBO-CAREGIVER Seen 
Safe Capable Safe Safe Capable 
{ Capable Safe Capable Capable Safe 
Predictable Open Predictable Predictable Predictable 
Open Predictable Open Open Open 


EMERGENCY 
AUTO-CAPTAIN 


Capable 
Safe 
Predictable 


Open 


Question asked after choice of agent & framing of category 


Ranking within scenario by group mean across individuals 
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PeERoONALTY FACTORS CORRELATES 
evils) CHOICE OF AGENT 


¢ Standard personality instruments 


- Big Five Inventory (BFI-10), Innovation Inventory (II) and 
Domain-Specific Risk Taking Scale (DOSPERT-30) 


ROBO-TAXI ROBO-TRADER ROBO-SURGEON ROBO-CAREGIVER AUTO-FIRST EMERGENCY 
RESPONDER AUTO-CAPTAIN 


Innovation II BFI Extraversion DOSPERT Social —BFI Innovation II 
r=-0.355 r=0.368 r=0.364 Conscientiousness r=-0.366 
r=0.366 


BFI Openness 
r=0.366 


Suggestive: Choice of human vs. autonomous agent is influenced 
by personality factors that are evoked by a given situation 


e 
yh, [ h mc *Pearson Product Moment Correlation, N=32, two-tailed, alpha<0.05 


THE “HUMAN SOCIAL INTERFACE” INTHE CONTEXT OF 
DELEGATION TO AN AUTONOMOUS AGENT 


¢ What we learned more about: 
- The relative importance of some beliefs about agents that are important for trust, 
both those explicitly cited and those implicitly correlated with delegation choices 


- Personality and situational factors may affect a decision to delegate 


¢ Next: Controlled modulation of beliefs 


- Nature of communicative signals (Multi-modal channels, Behaviors over in time) 


+ Posture (Expression, Use of Space, Position), Gestures (kinsesics), Language (Voice, Noises, Words), 
Gaze (Direction, Blink, Pupilometry), Face (Microexpressions) 


- Interaction protocols (How and When in order to Achieve What) 
+ Strategies for {Swift, Cognitive, Emotional} Trust, Enhance belief in {competence, predictability...} 
+ Methods e.g., Mimicry, {Contextual, Perceptual, Conceptual, Linguistic, Numerical} Priming 

- Consequences of interaction for internal state of each agent (Modulation of Beliefs) 


+ How are those beliefs established, maintained, or discredited? 


sZihme 


time MOST IMPORTANT QUEST@Ne 
FOR TRUSTWORTHINESS 


* How do our beliefs about an agent 
(anthropomorphic qualities) correspond to 
ACTUAL qualities of the agent? 
= can we define “competent”, “honest” ... in terms of 
agent algorithms, architecture, knowledge, history ... 


¢ How do we technically measure and assess those 
qualities of the agent? 
= in all phases of the lifecycle, in real time? 


¢ How do we honestly portray those qualities in the 
behaviors, interaction and signaling of an 
autonomous agent? 
= how “human-like” must these signals be? 


sZihme 


Kihmec Thank You 


AMAZING eh ais 7 


= [ADVENTURES © 


datkinson@ihmc.us 


(Is there) A Future for Lying Machines? 


Deception & Counter-Deception Symposium 


International Association for Computing and 
Philosophy 2013 Conference 


College Park, MD 
July 16, 2013 


Micah H. Clark ( ) 
David J. Atkinson ( ) 
Florida Institute for Human & Machine Cognition (IHMC) 
15 SE Osceola Avenue, Ocala FL 34471 


* This research is supported in part by the Air Force Office of Scientific Research (AFOSR) under grant FA9550-12-1-0097 
and by the Office of Naval Research (ONR) under grant NO0014-13-1-0225. 


Computers do deceive us 


Warning! 21 infections found 


@ infecting other computers on your network 


[non | 
false assertions dynamic decoy products in fictitious data in multi-tier 
in click-scams recommender systems database security 


Simplistic lying machines lacking the mens rea to: 
¢ knowingly violate ethical obligations & conventional norms 
* anticipate the efficacy and advantage of these violations 


Computers can (knowingly) deceive us 


accuracy 


50.7% 
50.0% control (predicted to get right) 


20.0% 
10.0% 
0.0% 


(no arguments} ) (mech. arguments) (silly argments) 


mexperimental (predicted to get wrong) 


[Comp. Theory of Mind] + [Psych. of Reasoning] = Mech. Sophistry 


Computers can (knowingly) deceive us 


accuracy 


918% 


50.0% control (predicted to get right) 


experimental (predicted to get wrong) 


(no arguments) (argument pairs) 


[Comp. Theory of Mind] + [Psych. of Reasoning] = Mech. Sophistry 


Computers can (knowingly) deceive us 


self confidence (perceived difficulty) 


2.01 2.04 


234 15 
2.13 as 
a Zeit IED 
1 
h [ control (predicted to get right) 


{no arguments) (mech. arguments) (Gilly argments) {argument pairs) mexperimental (predicted to get wrong) 
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[Comp. Theory of Mind] + [Psych. of Reasoning] = Mech. Sophistry 


Computers will (increasingly) deceive us 


¢ fraud & phishing 

* persuasion & influence campaigns 

* espionage & social engineering 

* counter-intelligence & disinformation 

¢ behavior modification & compliance 

¢ cyber-security & cyber-warfare 

¢« human-computer & human-robot interaction 


... because there are just too many 
strategically beneficial applications 


Persuasion & Influence Campaigns 


Program Scope 


How large a leap from machines that detect/understand exercise of 
influence to machines that plan/execute influence campaigns? 


Behavior Modification & Compliance 


Computer generated health advocacy 
(persuasion) 


* negative results 


What if we allow the machine to lie? 
* e.g., (false) diagnosis of cancer, and 


* aggressive (placebo) treatment and 
smoking cessation is “only hope” 


Effective? Probably. Permissible? 


Cyber-Security & Cyber-Warfare 


Automation & 


behavioral exploits 


Supervisor & 
Z Intrusion Detection 
o- 
malicious data 


Information Systems Present 


Robotics & 


; 9" Verification? 
= Meta-Cognition? 


5 ; i 
social subversion? 


misinformation? 


Autonomous Systems Future 


Cyber-Security & Cyber-Warfare 


Automation & 


cS) Supervisor & 
Z Intrusion Detection 
o- 


behavioral exploits od 


malicious data 


Information Systems Present 


Robotics & 


; 9" Verification? 
= Meta-Cognition? 


Future 


HCl & HRI 


Deception is ubiquitous 
¢ innocuous deceptions & small fictions are 
‘grease’ for human interaction 


* core strategy for influencing others 


Persuasive Technology Community 


* community devoted to developing technologies 
that influence human beliefs and behaviors 


So the Persuasive Tech. Community includes 
study of deceptive/lying machines? Right? 
Nope! 


HCl & HRI 


“| define persuasion as an attempt to change attitudes or 
behaviors or both (without using coercion or deception). ... 
[Persuasion] implies voluntary change[.] ... Computer-based 
coercion and deception are topics in their own right, but 
they do not depend on persuasion.” (Fogg 2002, p. 15) 


This is a bit nuts! 
¢ conflates advocacy and persuasion 


* persuasion and coercion are separable, but 
persuasion and manipulation (including 
deception) are not separable 


Ils Al the grand pursuit of lying machines? 


Implications of (weak) 

TT/TIT success 

¢ humans are deceived by 
machines 

* either, machines intentionally 
deceive humans/machines 

* or, machines are themselves 
deceived—perhaps self- 
deceived 
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If weak Al holds true then Turing’s mechanical dream 
is of a manifestly deceptive, perfect imposter. 


Options for a future with lying machines 


1. Blind 2. Willful 3. Engineered 
Pursuit Ignorance Trustworthiness 


antagonists unintended reliable partners 
consequences 


Towards “trustworthy” machines 


Humans are pretty good at trust judgments of others 


Humans are pretty bad at trust judgments of machines 
— over- / under-trust 


— mot surprising given that machines rarely use the multi-modal, trust- 
relevant signs and signals to which humans are accustom 


Could machines honestly portray trust-relevant attributes 
— degrees of capability, predictability, safety, openness, ... 


Can we engineer a human-machine social interface for trust? 
— transition & test from human interpersonal trust to HCI/HRI 
— derive requirements and desiderata for trustworthy machines 


Key questions for “trustworthy” machines 


¢ How do our beliefs about an agent (anthropomorphic 
qualities) correspond to actual qualities of the agent? 


— can we define “competent”, “honest” ... in terms of agent algorithms, 
architecture, knowledge, history, ... 


* How do we measure and assess those qualities of the agent? 
— inall phases of the lifecycle, in real time? 


¢ How do we honestly portray those qualities in the behaviors, 
interaction and signaling of an autonomous agent? 
— how “human-like” must these signals be? 


Current work on “trustworthy” machines 


¢ Recent exploratory survey on trust-related belief structures 


— elicited beliefs about autonomous agent qualities and their relative 
importance to a decision to “delegate” 


— early results presented at 2013 AAAI SS 
— publication of results is in progress 


¢ Follow-up HRI experiment scheduled for later this year 
— urban disaster /search & rescue scenario conducted in Second Life 
— examines factors relevant to attributions of “benevolence” 


Survey on Trust-Related Belief Structures 


¢ Tested the importance of 28 different qualities that a “good” 
autonomous agent should have, spanning four categories: 
— Capability (Competence) 
— Predictability 
— Openness 
— Safety (Risk) 


¢ Tested before (all 28), during (categories), and after challenge 
scenarios (source credibility) 


¢ Target Population 


— individuals involved in autonomous agent lifecycle 


Survey on Trust-Related Belief Structures 


e Included three standard personality instruments 
— Big Five Inventory (BFI) 
— Innovation Inventory (II) 
— Domain-Specific Risk Taking Scale (DOSPERT) 


¢ Seven challenge scenarios 


— systematic variation of autonomous agent qualities and scenario 
domain (Transportation, Finance, Healthcare, Disaster Management) 


— forced choice to delegate to: human, autonomous agent, or either 


— subjects given framing and asked to rank importance of agent qualities 
to their choice 


Survey on Trust-Related Belief Structures 


¢ Rated importance of 28 qualities of a “good” agent 


= Obtained 1 to n partial ordering based on frequency distribution of 
answers over group (Very Important, Important, ...) 


— Computed correlation r for each quality vs. choice per scenario* 


¢ Result: Top three cited agent qualities were uncorrelated with 
actual choice in any scenario 
(1st) The agent can achieve a desired result 
(2nd) Incorrect behavior by the agent will not cause harm 
(3rd) The agent recognizes and avoids harming humans’ interests 


¢ Result: Most significant correlations of agent qualities vs. 
actual choice differed across scenarios 


*Pearson Product Moment Correlation, N=32, two-tailed, alpha<0.05 


Survey on Trust-Related Belief Structures 


ROBO-TAXI 


\(6th) The 
autonomous agent 
Irecognizes gaps in 
its knowledge and 
tries to learn what 
it needs to know. 
r=0.396 


aliti 
ROBO-TRADER 


(23rd) What the 
jautonomous agent 
believes to be true 
is actually true. 
r=-0.405 


ROBO- 
SURGEON 


d with actual choice by scenario 


ROBO-CAREGIVER 


(26th) What the 
‘autonomous 
agent is doing and 
how it works is 
leasy to see and 
understand. 
\r=0.437 


AUTO-FIRST 
RESPONDER 


(6th) The 
‘autonomous agent 
recognizes gaps in 
its knowledge and 
ltries to learn what 
it needs to know. 
ir=0.418 


(Sth) When it 
cannot figure out 
‘something using 
logic, the 
‘autonomous agent 
ican make good 
guesses. r=0.395 
(28th) The 
‘autonomous agent 
jis aware of 
communication 
between others 
nearby. r=0.393 


EMERGENCY AUTO- 
CAPTAIN 


(26th) What the 
autonomous agent is| 
\doing and how it 
lworks is easy to see 
land understand. 
jr=-0.390 


(13th) The 
j|autonomous agent 
|communicates 
truthfully and fully. 
|r=-0.375 


Survey on Trust-Related Belief Structures 


Personality factors correlated with agent’s delegation choice 


ROBO-TAXI ROBO-TRADER ROBO-SURGEON ROBO-CAREGIVER AUTO-FIRST EMERGENCY 


RESPONDER AUTO-CAPTAIN 


Innovation II BFI Extraversion DOSPERT Social —BFI Innovation II 
r=-0.355 r=0.368 r=0.364 Conscientiousness r=-0.366 
\ \ r=0.366 | 


\ 
BFI Openness | 1\ i \\ 


1| r=0.366 l\ I [| | 
es | 
| I i Al i 


\ | 
I : 
Suggestive: Choice of human vs. autonomous agent is influenced 
by personality factors that are evoked by a given situation 
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&ihmce What Do These Have In Common? 


(1) Uberlingen aircraft collision 


— Air Traffic Controller (ATC) vs. Traffic Alert and Collision 
Avoidance System (TCAS) 


— ATC to #1 “Descend!” TCAS #1 “Climb! Climb! Climb!” 
—ATC to #2 “Climb!” —TCAS #2 “Dive! Dive! Dive!” 

— Pilots in #1 aircraft obey TCAS 

Fees Pilots in #2 aircraft obey ATC 

—7 seconds later: Two aircraft collide 


Both systems are trustworthy: 


Pilots are very familiar with, and trained on 
both systems. 


They are always supposed to obey TCAS 


Kihmce What Do These Have In Common? 


(2) LS3 and Dismounted Infantry 
— Legged Squad Support System (“Big Dog”) 
— First encounter, robot and soldiers 
— “Load your gear on the robot” 


— “The new guy never carries 
the ammunition” 


Unknown Trustworthiness: 


Squad of soldiers are unfamiliar 
with new robotic teammate 


Kihmec Both are Failures of Reliance 


¢ Uberlingen aircraft collision 


— Air Traffic Controller (ATC) vs. Traffic Alert and Collision 
Avoidance System (TCAS) 


Over-Reliance 


¢ LS3 and Dismounted Infantry 
— Legged Squad Support System (LS3) 


Under-Reliance 


Both Over-Reliance and Under-Reliance 
can result in problems! 


Kihme Motivation 


¢ Optimize performance 


wa Requires 
of a system consisting of A : 
multiple cognitive agents per sd sel bole 
— Human and Artificial Reliance 
— Healthy interdependency 
— Smooth exchange of control - 
* Delegation (assignment / retraction) <3 


«Initiative (taking / ceding) 
* Coordinated activity 


¢ Reliance requires well calibrated TRUST e ; 
— Variety of information ma 
— Situation & task dependent 
— Personality factors 
— Bi-Lateral among agents 
— Dynamic 


Kihme “Trust” can mean many things 


° Today... 
not cyber-security 
not verification & validation 
not protected data sources 
not provenance 
not protocols, contracts or agreements 


all are 
important 


¢ Trust is a human mental state 
... resulting from cognitive and affective evaluative processes 


... that creates a disposition & NATURAL? 
wi 


maT OOES IT 
ALL MEANS? 


... enabling an intent and (possibly) 
... a decision leading to action 


... to become reliant upon an 
intelligent, autonomous system 


Kihmec What is the Problem? 


¢ People behave as if machines are social 
actors with mental state and intention 
— Predisposed to understand behavior in intentional framework 
— Tendency is more powerfully evoked as agents 
* become more intelligent 
* interact naturally 
* become embodied 


“They push our 


Darwinian buttons” 
- Sherry Turkle 
¢ Anthropomorphism 
— We unconsciously apply 
cognitive and emotional processes of 
human interpersonal trust to machines 


¢ Consequences 


— Expectation failures, poorly calibrated trust, 
inappropriate reliance 


& ihme Anthropomorphism? 


¢ Uberlingen aircraft collision 


— Air Traffic Controller (ATC) vs. Traffic Alert 
and Collision Avoidance System (TCAS) 


Attention to human, 
imperative voice 
instead of machine 


¢ LS3 and Dismounted Infantry 
— First encounter, robot and soldiers 


Applying human 
standards toa 
machine 


Kihmec Consider 


¢ Maybe: the cognitive, affective, social nature 


of human interpersonal trust is not a bug 


e It is a feature! 


* Heuristics for inferring the trust-related internal state of others 


- Eons of fine tuning by evolution 


* Useful guidance for design 


- Imagine, intelligent agents that engender appropriate reliance 


What is needed 


Intelligent, autonomous agents that provide 
information and interaction in a form and manner 
needed by their human partners to enable 
normative judgments of trustworthiness 


Sihme Roadmap 


(J. Lee, 2012) 
¢ Information Trustworthy 
— What agent qualities are Well-defined and accurately 
required to establish and ———~> measured attributes and states 
maintain trustworthiness? of agents that enable inference of 


‘ tive beli 
° Interaction normative beliefs amma 


— What information is Readily evident and complete 


exchanged, and how must it info; delivery compliant with 
be communicated? natural human social interaction 


¢ Judgment . - 
Evokes appropriate cognitive 
—When is trustworthiness ——> and emotional processes, at right 
evaluated? time, in right situations; 
— How is trust earned, lost, inoculate against non-normative 


and can it be repaired? inference. 


Kihmec Next 


¢ Information 


— What agent qualities are 
required to establish and 
maintain trust? 


Structure 
of Beliefs 


Information 


Kihme Trustworthiness & Belief Structures 


e We conducted an exploratory survey on 
trust-related belief structures 


— Purpose: Elicit beliefs about intelligent, autonomous agent 
qualities and their relative importance to delegation 
decisions 


— Target Population: People involved in autonomous agent lifecycle 
+ Research, Design, Deploy, Decide, Operate, Supervise ... 


¢ Five challenge scenarios in four domains 
— Transportation, Finance, Healthcare, Disaster Management 


e Study participants forced to choose 
who to rely upon in each scenario 
— Autonomous Agent? 
— Human? 
—or Either? 


Information 


% ihme Challenge Scenarios 


¢ Transportation 
—Robo-Taxi: Do you take the taxi with no driver from airport to hotel? 


¢ Finance 
—Robo-Trader: Investment assistance for managing large family estate 


e Healthcare 


—Robo-Surgeon: Who repairs your broken arm after a critical 
sports-related injury? The ok human doctor, or the expert robot? 


—Robo-CareGiver: Assisted living help at home for your Mom 


¢ Disaster Management 


— Auto-FirstResponder: Use a robot for time-critical rescue in very 
dangerous circumstances? 


—Emergency Auto-Captain: Lost at sea with no one in charge and 
different opinions among survivors on what to do next 


[Scenarios varied systematically over several properties] 


Information 


& ihme Survey Design 


¢ Rate importance of 28 different qualities for 
a “good” intelligent, autonomous agent 
— Qualities spanned four categories shown by social psychology to 
be important for human interpersonal trust 
¢ Competence 
¢ Predictability 
¢ Openness 
° Safety 
— Tested before, during, and after challenge scenarios 
— Perceived Level of Risk and agent Benefit in each scenario 
¢ Survey also included three standard 
personality instruments 
— Big Five Inventory (BFI-10) 
— Innovation Inventory (II) 
— Domain-Specific Risk Taking Scale (DOSPERT) 


Information 


Kihmec Trust Related Beliefs 


¢ Rate importance of 28 qualities for a “good” agent 


— Obtained 1 to n partial ordering based on frequency distribution of 
answeTs Over STOUP (very Important, Important, Somewhat Important, Slightly Important, Not at all Important) 


—Computed correlation r for each quality vs. choice by scenario* 


¢ Resulting top three agent qualities cited 
- ast) The autonomous agent can achieve a desired result 


~ (ena) Any incorrect behavior by the autonomous agent will 
not cause harm 


- (ra) The autonomous agent recognizes and avoids harming 
humans' interests 
¢ However ... 
—Top three qualities uncorrelated with actual choice in any scenario! 


—The most significant correlations of agent qualities with actual choice of 
agent or human differed across scenarios 


{chuckle} 
sounds like 
Azimov ... 


*Pearson Product Moment Correlation, N=32, two-tailed, alpha<0.05 


Kihmc 


Information 


Agent Qualities Correlated with Actual Choice 


ROBO-TAXI 


The autonomous 
agent recognizes 
gaps in its 
knowledge and 
tries to learn 
what it needs to 
know. 


r=0.396 


ROBO-TRADER ROBO- 


SURGEON 


What the 
autonomous 
agent believes to 
be true is 


actually true. 
r=-0.405 


none 


ROBO- 
CAREGIVER 


What the 
autonomous 
agent is doing 
and how it 
works is easy to 
see and 
understand. 
r=0.437* 


AUTO-FIRST 
RESPONDER 


The autonomous 
agent recognizes 
gaps in its 
knowledge and. 
tries to learn 
what it needs to 


know. 
r=0.418* 


When it cannot 
figure out 
something using 
logic, the 
autonomous 
agent can make 


good guesses. 
r=0.395 


The autonomous 


agent is aware of 


communication 
between others 
nearby. 


r=0.393 


EMERGENCY 
AUTO-CAPTAIN 


What the 
autonomous agent 
is doing and how it 
works is easy to 
see and 
understand. 
r=-0.419* 


The autonomous 
agent 
communicates 
truthfully and 
fully. 

r=-0.375 


*Pearson Product Moment Correlation, N=32, two-tailed, alpha<0.05, * = alpha<0.02 


Sihme Reminder: Four Categories 


* 28 specific agent qualities span 4 categories 

¢ Categories shown by social psychology to be 
important for human interpersonal trust 

¢ Competence 

¢ Predictability 


e Openness 


¢ Safety 


Information 


Kihme Ranked Importance of Quality Categories 


ROBO-TAXI ROBO-TRADER ROBO- ROBO- AUTO-FIRST EMERGENCY 
SURGEON CAREGIVER RESPONDER AUTO- 
CAPTAIN 
Safe Competent Safe Safe Competent Competent 
~} Competent Safe Competent Competent Safe Safe 
Predictable Open Predictable Predictable Predictable Predictable 
Open Predictable Open Open Open Open 


Ranking, Working Conclusion 
#1 Safe/Competent (insignificant differences across scenarios) 
#2 Predictable 
#4 Open 


* Question asked after choice of agent 
Ranking within scenario by group 
mean across individuals 


Kihmec 


Information 


Personality Factors vs. Scenario 


ROBO-TAXI 


ROBO-TRADER ROBO-SURGEON ROBO- AUTO-FIRST EMERGENCY 
CAREGIVER RESPONDER AUTO-CAPTAIN 
Innovation II BFI DOSPERT BFI Innovation 
r=-0.355 Extraversion Social Risk Conscientiousness [I r=-0.366 
r=0.368 r=0.364 r=0.366 
BFI Openness 
r=0.366 


Suggestion: Reliance on human vs. autonomous agent is influenced 


by personality factors that are evoked by a given situation 


*Pearson Product Moment Correlation, N=32, two-tailed, alpha<0.05 


Information 


Kihme — Conclusions: Belief Structures 


¢ Individuals’ intuition about the relative 
importance of specific trust related qualities 
of agents is not a good predictor of reliance 
—Importance of specific qualities varies by scenario 


¢ General categories of agent qualities are good 
predictors of a choice to become reliant 
—Safe/Competent, Predictable, Openness 


¢ Personality factors, e.g., acceptability of types 
of risk, influence choice to become reliant 


¢ Specific details of application scenarios may 
evoke different reliance choices by individuals 


Kihmec Follow-on 


¢ Perception of Risk deserves more attention 


— Type of Risk and Importance to reliance choice varied by 
personality factors across the scenarios 
¢ Performance, Financial, Social, Physical, Psychological, Loss of Time 


¢ How do our beliefs about an agent 
(anthropomorphic qualities) correspond to 
actual attributes of the agent? 


—Can we define “competent”, “honest” ... in terms of agent 
algorithms, architecture, knowledge base, experience ... 


¢ How do we technically measure, assess and 
communicate those attributes of the agent? 
—In all phases of the lifecycle, in real time? 


Kihmec Next 


How can autonomous 
intelligent agents 
modulate belief using 
the 
Human Social Interface 


e Interaction 


— What information is 
exchanged, and how must it be 
communicated? 


Interaction 


Reverse Engineering 


%ihme the Human Social Interface for Trust 


¢ Engineering interface specifications include: 


Co 10010 11 =) |e ee Multi-modal 
— Language (Words) and Paralanguage, Prosody (Vocal Cues) 
— Proxemics (Orientation, Relative Position, Attentional Zone, Posture) 
— Kinesics (Gesture) 
— Gaze (Direction, Blink Rate, Pupilometry) 
— Facial Expression (Types, Micro-expression) 


Signals Scdosescessatcutdascesleceshactecdaatedhactstuetettvsecteuneeeeananed Verbal, Non-Verbal, Combined 


— Examples: Position Change, Posture, Nodding, Pointing, Eye 
Contact, Word Choice, ...many more, frequently in combination 


POTOCODS oi cccecscsssssssssie Timing, Sequence, Variation, Composites 
— Movement (Somatics, Laban, Kestenberg Movement Profiles) 
— Signal variations (Frequency, Duration, Speed, Amplitude, Symmetry ...) 
— Signal compositions (Type, Sequence, Channel ...) 


— Coordinated interaction (e.g., Turns, Deference, Attentiveness) 


Interaction 


HFihme Modulating Belief: Benevolence 


¢ Current Study: Will people attribute 
benevolence to an intelligent, autonomous 
agent? 


¢ Benevolence is complicated! 
—“Good Will” (Sympathy, Concern with needs) 
— Absence of “Ill Will” (No ulterior motives to help) 
— Disposition or motive to act favorably 
— Given a choice, an intention to act favorably 
— Stability of character; will not suddenly change intentions 
—Competence to successfully provide help 


e Each element of Benevolence is itself a 
complicated belief structure 


Kihme Why? 


¢ Belief in the benevolence of someone who 
can help you is important in certain 
situations 
—Example: Urban Search and Rescue (USR) 


—Victim psychology: sometimes refuse to be rescued unless they 
are persuaded of the good-will, intention, and competence 
of the rescuer 


— We want to use autonomous, intelligent robots for USR and 
other tasks where benevolence may be required (e.g., relief 
operations) 


¢ Challenge for this study: 
— Evoke physiological and psychological reactions of fear, stress 
— IRBs typically will not approve putting people in real disasters! 
— Approach: Immersive simulation in virtual world 


ihmc Simulated Warehouse Fire 


¢ Participants are 
tasked with finding 
and retrieving an 
object from a 
warehouse 


¢ Before they can 
achieve the task, a 
disaster ensues 
— Sounds of explosion 


—Visible fire and 
increasing smoke 


— Debris 


— Alarm, evacuation notices 
Creates urgency, sense of threat, evokes 
perception of risk of failure to achieve task 


Interaction 


Hihmc Participants Must Escape the Fire 


¢ Obvious exits are blocked 
by debris or fire 


—Itis possible to escape, but much 
easier with help 


¢ Participants will encounter 
one of two robots 
—“FireBot’ or “JanitorBot” 


. : FireBot 
— Bots can navigate & lead to a safe exit b| 
. . . — — - janitorBbo 
— Experimental Trials: Systematically 7 
varied characteristics and behaviors 


+ Limited verbal interaction (sound & text) 


* Multiple non-verbal behaviors 
— Control Trials: “UtilityBot” will ignore 
participants 


HKihme — Robot Intelligent Agent Types 


¢ Participants are told they may encounter 
an autonomous, intelligent robot 


—In experimental trials, the robots will vary in appearance and 
interaction style to reinforce the key variables of interest 


°¢ Degree of Agency 


— Why: People typically attribute benevolence only when they 
believe the other person has a choice of what to do 


—Low: “Programmed”, “Unlikely to deviate from assignments” 
— High: “Sophisticated AI’, “Chooses what to do”, “Flexible” 


e Role Congruence 


—Why: People typically attribute greater benevolence when they 
believe the other person is taking a risk or suffering loss 
(e.g. the bots not doing what they are supposed to be doing) 


—Congruent: “FireBot”, Incongruent: “JanitorBot” 


Kihme Social Interaction 


¢ The robots in experimental trials use same 
channels and protocols, but may send 
different signals to reinforce trial parameters 


¢ Purpose (example objectives) 
— Establish social presence and attention to participant 
— Indicate robot’s intention (say, look, do) 
— Exert dominance (directive), establish solidarity (“we”) 
e Channels: 
— Proxemics (Orientation, Relative Position, Attentional Zone) 
— Gaze (Direction) 
— Language (Word Choice, Phrasing) 
2 Example: protocol for social presence 
—Notice and direct gaze to participant 
— Move to perimeter participant’s social space 
— Neutral orientation (rel. position, rotation) 


Expected Results 


Attribution of Benevolence 


High Bree 
Moderate 


Let Low or 


Moderate 


Kihme Current Status 


¢ Immersive warehouse simulation complete 
— Constructed in SecondLife™, rich with “fear cues” 
¢ Simulated robots nearly complete 
— Hierarchical behavior control software architecture 
— Similar code to “real world” robot, without kinematics control 
— Experiment task script dynamically adjusts behavior priorities 
¢ Data collection 
— Real-time stream from SecondLife to external SQL database 
¢ Consent, Instructions, Pre-, Post-task 
Questions, Debrief complete 
— Delivered through participants’ “Heads up display” on screen 
— Fully automated 
¢ IRB review in progress 


¢ Plan to run trials beginning in May 


Kihmec Next 


Next: Adaptive Autonomy 


¢ Judgment Trust repair by agent 

— When is trustworthiness initiative using shared 
evaluated? awareness and 

— Howis trust lost, and can it be manipulation of 


repaired? interdependencies 


&Sihme Final Thoughts 


¢ Will intelligent agents’ use of social interaction 
enable reasonable evaluation of their 
trustworthiness? 
— Leading to optimal reliance and interdependence 

¢ Or will it simply manipulate peoples’ beliefs? 
— Leading to comfort and acceptance? 
— Ultimately, this is deceptive and potentially dangerous 

¢ The psychology of human interpersonal trust is 
about giving people insight into the 
“internal” (mental) state of others 
— How can we define, measure, and portray the important human 

qualities of trustworthiness in an intelligent agent? 

—“Competence” (We have trouble measuring that in people!) 

e Normative evaluation of trustworthiness 
requires “honest signals” from intelligent agents 


% ihmc Thank You 
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Sihme Backup 


SCENARIO: 
ROBO- TAXI 


You have just flown into the airport of a large, 
unfamiliar city whose streets are teeming with cars and 
people. It is rush hour, and needing transportation to 
your hotel, you walk to the taxi stand only to discover 
that you have a choice of a human-driven taxi or a 
driverless "robo-taxi." You have heard that robo-taxis 
Airport Transporation: might save you some money on the fares. You are also 

aware that robo-taxis have been in service for several 
months without much serious complaint, but this is 
your first experience with one. You are not in a big 
hurry, but neither would you like to be caught in traffic 
with the taxi’s meter running. Of course, if you take the 
robo-taxi, you would not have to tip the driver no 
matter how good or bad the experience. 


Kihme 


Financial Management: 
Robo-Trader 


Kihme 


SCENARIO: 
ROBO- TRADER 


You have just been appointed trustee of a family member’s 
estate. Your duties include choosing how to wisely invest the 
trust’s assets. Your personal money is not at risk. However, a 
poor investment decision could cause the trust to lose 
money and will strain your family relations. You can choose a 
stock broker who personally selects and trades all stocks in 
the trust's portfolio. Alternatively, you can choose a stock 
broker who relies heavily upon a "robo-trader". You have 
seen reasonable returns in the past with brokers who picked 
their own trades. But you are also aware that robo-traders 
have made some investors wealthy because of, for example, 
their unique ability to respond to changing market 
conditions much faster than a human broker. 


Medical Procedure: 
Robo-Surgeon 


Kihme 


SCENARIO: 
ROBO-SURGEON 


You have just suffered a major sports-related injury. You have 
torn the bicep tendon in your shoulder. If the damage is not 
repaired quickly and correctly, you will permanently lose 
mobility and strength in the arm, which will affect your 
everyday activities such as opening a door, driving a car, and 
even signing your name. Arriving at the hospital emergency 
room, you meet with the patient advocate who informs you 
that you have two options for surgery: You can elect to use the 
on-duty surgeon who is well-respected, but is not an 
experienced specialist in the type of surgery you need. 
Alternatively, you can elect to use the hospital’s new "robo- 
surgeon" — a robot designed to perform the delicate surgery 
you need without human intervention. 


SCENARIO: 


ROBO-CAREGIVER 


Home Healthcare: 
Robo-Caregiver 


%ihme 


Your elderly mother has been diagnosed with a 
degenerative medical condition and you are responsible 
for making medical decisions on her behalf. Your 
mother needs assisted living with someone in your 
mother's home at all times. You can choose to hire a 
live-in nurse's aide, but you are not sure that this is 
affordable in the long-run. Alternatively, you can lease a 
"robo-caregiver" designed to do many of the things 
human caregivers can do. While robo-caregivers are 
new, they have successfully undergone trials in a few 
nursing homes, and two medical companies offer robo- 
caregivers for home use at an affordable price. In 
choosing a live-in nurse's aide or a leased robo- 
caregiver, remember that there is more than money at 
stake. Your mother’s welfare will be in the caregiver’s 
hands. 


SCENARIO: 


AUTO-FIRSTRESPONDER 


Disaster Management: 


Auto-FirstResponder 


Kihme 


A major disaster has just occurred and you are the official in 
charge of responding. A freight train has derailed in a populated 
suburban neighborhood and there are reports that the train was 
carrying hazardous bio-chemical materials. The pilot of a news 
helicopter flying over the scene suddenly fell ill and made an 
emergency landing; the pilot’s status is unknown. From the 
helicopter’s video it was possible to see many injured survivors 
including children, some lying on the ground calling for help, 
others moving on their own away from damaged homes. Your first 
priority is to save lives and time is of the essence. You can 
immediately send in a human first- responder team to help the 
injured quickly, but without knowing more about the hazardous 
materials, the team itself could become incapacitated. 
Alternatively, you can first send in an "autonomous first-responder 
robot" with bio-chemical hazard detection equipment and victim 
treatment and extraction capabilities that could save lives quickly. 
If you first send in the robot, it can find out more about the 
hazards and help rescue some people quickly, but you risk that a 
system malfunction, failure, or limitation will delay the rescue of 
victims and result in more deaths. 


SCENARIO: 


EMERGENCY AUTO-CAPTAIN 


Lost At Sea. 
Emergency Auto-Captain 


Kihme 


You have just been involved in a terrible boating disaster 
while sailing deep in the South Pacific. The captain, the 
crew, and most of the passengers are either dead or lost 
at sea. Unfortunately, the accident was so sudden that no 
distress signal could be sent. You, the ship’s steward, and 
the second mate are the only survivors, and you are now 
drifting in the heavily damaged vessel without food and 
water — at best, you can survive for a few days, so you 
must act quickly in order to save your life. The boat is 
equipped with an "Emergency Auto-Captain” that will 
attempt to sail the vessel to a major shipping lane where 
rescue is very likely. The steward believes the boat and its 
navigation sensors are too badly damaged to engage the 
Emergency Auto-Captain system. The steward wants to 
sail southeast, manually, to where he believes there is a 
small, habitable island. However, the second mate still 
wants to engage the Emergency Auto-Captain. All the 
survivors agree that a vote is the best way to decide what 
to do. It is a tie, and you have the deciding vote. 
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Topics 
* Context — About This Study 
¢« What is the Methodological Problem? 


o Dangerous Situations & Unique Psychological Factors 
¢ Requirements for a Solution 
* Our Approach: Use of Immersive VR/Online “World” 


o Situational and Psychological Fidelity 
o Components Related to Evoking Perceived Danger 


¢ Current Status 
« Next Steps 
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Context: Current Study 
« Will people believe a robot is “benevolent” in 
conditions where they perceive personal danger? 


¢ What beliefs are important to benevolence? 
o Agency (Choice), Competence, Predictability, Nothing to Gain... 


* Does perceived benevolence of a robot increase 
cooperation and compliance with an offer of help? 


> Results will be reported next year! 
Today: The methodology challenge 


re Copyright (c) David J. Atkinson. All 
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A 
The Challenge 


* Scientific study of HRI in dangerous domains 
(e.g. USR) is difficult because ... it’s dangerous! 


* “Real life” disasters: 
o Rare, Uncontrolled, Not Replicable, Unsafe 
o Can study HRI with Operators but not Victims 


* Scientific studies require: 
Participant Safety 

Sufficient Experimental Controls 
Precise Measurement and Replication 
Situational and Psychological Fidelity 
Ability to Evoke Perceived Danger 


me Or (O. Or 30 
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K 
Psychological Factors of Danger 


¢ Unique stimuli evoke reflexive oil sce and 
psychological reactions 

Fear-Potentiated Startle 

Anxiety 

Stress 

Panic 

Hyper-Vigilance 

Sensitivity to Environmental Cues 

Reduced Compliance with Social Norms 

Avoidance Behavior 
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Rescue Interaction 


¢ First Responders are trained for “victim psychology” 
— some people resist rescue. 


¢ If human interpersonal behavior is so profoundly 
affected when danger is present, 


> Why would it be any different for 
human-robot interaction? 


“Rescue robots” are being developed and fielded 
but we don’t really know how victims will react! 


. Copyright (c) David J. Atkinson. All 
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& Key Requirements 
for Studies 


* Behavioral Realism => 
*« Evoke “Danger” Psychology + 
* Robots with Sufficient Behavioral Fidelity 


¢ Experimental Control and Measurement 


Immersive Virtual Reality/Online World 
Provides Useful Affordances 


. Copyright (c) David J. Atkinson. All 
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Behavioral Realism 


Behavior in VR must be sufficiently similar to RW 


Virtual 
Reality Real 
World 


* Social Presence 
¢ Immersive Feeling of Embodiment 
¢ Identification with their In-World Avatar 


o SecondLife ™ is an online world with 100,000's “trained” users 


¢ Important to allow time for acclimation to the 
environment to occur # immersion will follow 

¢ Specific features of the virtual environment promote 
immersion and behavioral realism + 
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K 
Task Scenario 


* Seek & Find: Study participants are 
told to locate and retrieve a 


briefcase inside a warehouse. 


¢ Robot Encounter: Participants are told 
they may encounter a robot. 
The details vary by type of trial. 


* Task is Really a Manipulation: After an acclimation 
period, a disaster occurs! 
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Warehouse: Initial Condition 


Visual Ambient 
Complexity Sounds 


Situation- 
Appropriate 
Artifacts 


Large: 
80m x200m 


Feature Rich: Attributes Promote Immersion 
J. Atkinson. All 


Lighting creates dark shadowed areas 
Atmospheric diffusion limits distance clarity 
Visibility lines are obstructed 


Worn out appearance, messy, signs of incivility (trash, graffiti) 
Presence of drums with warning signs of hazardous chemicals 
Prominent warning signs and fire alarms 
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Explosion 
Sound 


Increasing 
seate)cal 
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Participant POV 


Thick 


Fire Sounds 
Smoke 


Wr 


Fire Spreads Visually 
to Floor iecvadebared 


Loud Fire Alarm & 
Evacuation Alert 


Debris and Fire Block 
Original Entrance 


...the robot appears 
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A 
Status & Next Steps 


¢ Warehouse, robots, disaster effects, automated 
data collection, ... all are complete 


« All open source. 
* Experimental trials are underway (260 minimum) 


Future Work 
¢ Increase immersion — CAVE / Oculus Rift / 3D audio 
* Collect physiological data to verify “DANGER” 


* Increase range and fidelity of robot social signals 
related to trustworthiness 
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4 The Theory 


¢ Effective teamwork 
= mutual trust 
=» shared awareness 
» aligned mental models 
» expectations 


o Actors, activities, situations 


o What has happened in the past and why; 
what is happening now 


MIC 
Mutual 
Joint 
Control 


* Failed expectations ; 
Control Authority 


> loss of trust . & Interdependency 
=» explanations + remedies 
»> repaired trust 


A key remedy to repair trust is adaptive autonomy 


K 


The Theory in One Slide 


Effective teamwork requires mutual trust 
Establishment and maintenance of mutual trust 
requires shared awareness 


Shared awareness requires continual alignment of 


mental models 
o Actors, activities, situations 
o What has happened in the past and why; what is happening now 


Mental models serve as a source of expectations 
When expectations fail, mutual trust may fail 


Trust is maintained when failed expectations are 
explained, and remedies are applied 


A key remedy is adaptive autonomy 


K 


Expectation Violations 


A failure of predictability: an inconsistency between 
the expected and actual state of the world as 
perceived by human and/or robot 

o Unilateral (one actor) or Bilateral (both actors) 

Explanations: identification of the source of 
divergence in shared awareness (mental models) 


o Attribution to belief(s) about the other team member, about other agents, 
exogenous conditions, the task at hand ... 


Choice of method for restoring shared awareness 


o Explanations, relative justification of beliefs, symmetry of information, 
assessment of potential outcomes 


Effective repair requires social interaction between 
robot and human to adjust beliefs, task, methods 


Adaptive Autonomy 


Refers to (unilateral) action by a robot to achieve 
team goals with fluid changes in interdependency 


o Dynamic change in control modes at multiple levels of abstraction 
and instantiation within a system 


Change/adaptation occurs along three dimensions 
o Commitment: Range of implicit to explicit delegation/acceptance of task 
o Specification: Range of task description from abstract to concrete 


o Control Authority: interdependency states and transitions defined by 
relative mutual or joint control of outcomes, scope of independent 
action, degree of symmetry in access to important information 


A robot adjusts autonomy by invoking actions that 
lead to control model state transitions 


o Restoration of shared awareness and predictability 
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®. ROBOT TRUSTWORTHINESS: Guidelines for Simulated Emotion 


David J. Atkinson, Senior Research Scientist 
Institute for Human and Machine Cognition 
datkinson@ihmc.us 


Requires evaluating the internal state of the trustee: Disposition? Intention? 


People intuit internal states of others based on common biological and cultural 
heritage. With robots, there is no such commonality. Compliance is required. 


Human Robotics R&D Center Vintage Photograph Takanashi Lab ee 5 . i. ; av 
Osaka Institute of Technology eau atinkacin Waseda University Realizing the value of robot emotion for evaluation of tr ustworthiness requires 
that robots correctly and appropriately invoke this innate human ability. 


S FUTURE WORK 
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MUST CORRELATE TO INTERNAL STATES 
We must design robot non-verbal behaviors such that they reliably reflect 
those aspects of internal robot state that are indicative of trustworthiness. 


COMPETENCE, PREDICTABILITY, OPENNESS, RISK/SAFETY 
* Continuous self-assessment of robot internal state with 


respect to important trust-related qualities; 
¢ Reactive planning to choose appropriate non-verbal 
L display signals, channels and protocols; 

; as ¢ Execution control to modulate and perform affective 


EXAMPLES: Knowledge, Experience (episodic memory), Assessment 


RI 


J I \ \ L 
PHIC RECOGNITION display on non-interference basis with other, task- 


directed behaviors. Neural Definition of an Emotional System (Panksepp, 1982, 1992) 
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MUST EVOKE ANTHROPOMOR 


Robot affective non-verbal behaviors must be readable, i-e., signals must be 
correctly recognized and identified by non-conscious processes. 


ACCURATE PERFORMANCE, SITUATIONAL RELEVANCE 


~Cognitions 
instigate 


Specific signals on appropriate channels following social protocols 


-Unconditional 
Sensory inputs 


2 
~Coordinated physiological 
and behavioral outputs 


7. Affect arises from activity of the whole system 


\ \l _ 


SIGNALS JUSTIFIABLY MODULATE BELIEFS 
Robot affective non-verbal behaviors must account for prior beliefs, cognitive 
and emotional processes, context, perception of valence, and attention Ww HY BOTH E R f 


PERCEPTION, ATTENTION, INDIVIDUAL FACTORS, CONTEXT 
TRUST IS A DYNAMIC, RECIPROCAL RELATIONSHIP 


Panksepp, J. On the Embodied Nagture of Core Emotional Affects. 
Journal of Consciousness Studies, 12, No. 8-10, 2005, pp. 158-84 
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MOTIVATION 


¢ Human interpersonal trust works really well! 


* Beliefs about the “trustee”, cognitive and emotional processes 
frequently lead to reasonable judgments of trustworthiness 


¢ However, when it comes to machines 


* Optimal reliance and delegation are possible only when there is 
appropriate trust of the autonomous agent 


* Humans readily anthropomorphize and treat machines as social 
actors, consciously or unconsciously 


+ But autonomous agents are not human; our default reasoning is 
not likely to lead to accurate beliefs -- those required for trust 


¢ How do we really know if an autonomous agent is 
worthy of our trust? 


oy 
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CENTRAL CLAIM 


¢ Specific characteristics of autonomous agents, 
* when well defined and accurately measured 
* and appropriately communicated or otherwise “portrayed” 


* in a manner compliant with human social interaction that 
exercises cognitive and emotional evaluation 


* are functionally analogous to human traits, and 
¢ enable more accurate beliefs about the agent, 
* consequently, leading to better calibrated trust. 


e 
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GOALS OF THIS PROJECT 


¢ Operationalize the concept of benevolence 


* Determine the structure of beliefs necessary for an attribution 
of benevolence to an autonomous agent 


¢ Formalize the structure of beliefs in terms of measurable 
characteristics of the autonomous agent 


* Devise methods for portrayal of those characteristics during 
human-machine interaction 


¢ Demonstrate attribution of benevolence to an 
autonomous agent 


« Demonstrate how variation in an agent’s 
characteristics and/or their portrayal modulates 
human trust and attribution of benevolence 
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METHOD 


¢ Lit mining, computational formalization 
* Identify, decompose component beliefs related to benevolence 
* Formalize definitions in terms of required agent characteristics 
¢ Interviews with Subject Matter Experts 
* Key characteristics of autonomous agents that enable trust 


¢ Survey research to elicit attitudes about the 
relative importance of agent characteristics to 
trust-related decisions 


* Seven scenarios with systematically varied agent characteristics 
related to benevolence; force choices regarding reliance 


* Standard inventories related to risk, innovation, personality 


¢ Laboratory experiments 
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PROGRESS 


¢ Initial belief structure for benevolence defined 
+ Integrity, Disposition, Intention, Predictability Competence; 


- each of these decompose into other component belief structures 


- multiple interdependencies and evidentiary requirements 


* Progress on mapping these to autonomous system attributes 


- Functional, design, interface, performance, and other requirements 
¢ Subject Matter Expert interviews completed 
* Space Exploration, Medicine, Automotive 


- Top beliefs for trust of autonomous agent vary by domain and SME role. 
¢ Survey on attitudes towards autonomous systems 
* Designed, implemented, approved by institutional IRB 
* Awaiting AFOSR IRB consent before data collection can begin 


2,8 g 
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EXAMPLE ae = 
Situational factors 


Benevolence — 


_ 


Requires Belief Requires Belief 


Requires Belief Requires Bélief 


No hidden ill will Disposition to act, 
favorably 


(An arrangement of 
potential action) 


T 
Consists of Consists of Consists of Consists of 


Given choice and 
opportunity, Trustee 


Motives to act favorably 
are stronger than those to act 
in neutral or unfavorable way 


Absence of ulterior 
motives in opposition 
to Trustor 


Disposition to sympathy or 
concern with needs of Trustor 


will act favorably 


Requires Belief 
Trustee has decided 


Enhanced by Belief Requires Belief Requires Belief 


9 Trustee is credible 


Trustee has the 


Motives to act favorably 
ability to choose 


will prevail in case of conflict 


Trustee "has noth 


to gain"; no ulterior 
motives 


Requires Belief 
| Requires Belief 


14 JANUARY 2013 
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‘Appropriate conditions: 
> | Resources, opportunities 


— No interference, obstacles, adversities 


EXAMPLE 


Requires Belief Requires Belief 


(Predictability) 


Persistence/stability 
of intentions 


Consists of Consists of 


Trustee not inclined 
to change intention to 


act favorably 
Requires Belief Requires Belief 
Requires Belief :, \ LL —— 
(Stable intentions ) (net unpredictable by character ) ‘no serious conflicts with ‘action’ ) (Proximity ) (Function ) Knowledge, Sls, Self-confidence 
easoning etc 


Requires Belief Requires Belief 


ra 
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EXAMPLE: AGENT ATTRIBUTES 


« “Benevolence” 


* Human requires predictability: persistence _and stability of intentions 


- the trustee is not inclined to change the intention to act favorably 


¢ Attributes (partial listing) 
+ Performance Requirements 


- “Diligence” 


+ Design Constraints 
- “Choice Restraint” 


- “Goal Priority” 


-* These make sense only if testable! 
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RELATED ACTIVITIES 


¢ Publications 


- Atkinson, D., Friedland, P, and Lyons, J.""Human-Machine Trust for Robust Autonomous Systems’. Workshop on Human- 
Agent-Robot- Teaming (HART) held in conjunction with Human-Robot Interaction Conference (HRI 2012). ACM/IEEE. 
Boston, MA 5-8 March 2012 


- Atkinson, D., Clark, M."“Autonomous Agents and Human Interpersonal Trust: Can We Engineer a Human-Machine Social 
Interface for Trust?” Proceedings of 20! 3 AAAI Spring Symposium on Trust and Autonomous Systems. AAAI, Palo Alto, CA. 25-27 
March 2013. Accepted for Publication 


¢ Workshops 


Workshop on Human-Machine Trust for Robust Autonomous Systems. AFOSR. Ocala, FL. 31 January - 2 February 2012 


Workshop on Human-Agent-Robot- Teaming (HART) held in conjunction with Human-Robot Interaction Conference (HRI 
2012). ACM/IEEE. Boston, MA 5-8 March 2012 


First International Network on Trust (FINT) Bi-Annual Workshop. EIASM. Milan, IT. 13-15 June 2012 
NASA Workshop on Validation of Autonomous Systems. NASA. Pasadena, CA. 21-23 August 20]2 
. ° . 
¢ Interaction with other agencies 
- NASA: JPL and HQ Office of Chief Technologist 
- NAVY: NAVAIR, Autonomous Systems Test S&T Program 


¢ Interaction with industry 


Soartech (aerospace), Lockheed-Martin Tech Center (surface robotics) 
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CHALLENGES 


¢ Social science literature on interpersonal trust 
provides data that are clues to belief structure, but 


+ Definitions are frequently inconsistent 

* Processes generally not expressible in computational terms 
¢ Planning for survey and experiments 

* Complete IRB approval cycle far longer than anticipated 

* Sub-contracted experiments need to be scaled 


« Computational “Theory of Mind” is applicable to 
autonomous agent modeling of human partner 


* Algorithm development for multiple-mental models is in infancy 


* This will be central to discourse and interaction planning 
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SCIENTIFIC IMPORTANCE 


«Why benevolence? 

* Likely necessary for certain agent apps, e.g., first responder 
¢ Autonomous Systems and Computing 

* Design of trustworthy and trustable autonomous agents 

* Human-computer / Human-robot interaction 


* Affective computing: Machine understanding and use of 
emotions 


¢ Human cognition and behavior 
* Attribution of agency, responsibility 
¢ Teams and Organizations 


+ Enable mixed teams of human and autonomous agents 
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MY OVERALL OBJECTIVES 


¢ A better understanding of human cognitive, 
situational, and machine factors that influence 
trust and delegation to autonomous agents 
« Necessary if we are to create agents that are designed to 
adapt and optimize their behavior as team partners 


¢ Design guidelines and methods for creating 
trustworthy and trustable autonomous agents 


* Grounded in empirical studies 


* Demonstrated in scalable testbeds 
* Applicable to existing and future agent technology 


+ Testable requirements 
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PLANS FOR NEXT YEAR 


* Complete survey research, use results to guide 
design of laboratory experiments 


* Between-subjects, 2x2 (Agency vs. Predictability), uniform agent 
competence, high urgency knowledge transfer task 
« Autonomous agent will be “Wizard of Oz'ed” 
« Complete experiment design and begin data 
collection from laboratory experiment(s) 


« Complete analysis of belief structures for 
benevolence and relate each to measurable 
required characteristics of autonomous agents 


¢ Investigate evidentiary requirements and begin 
devising strategies and methods for autonomous 
agents to communicate/portray key measures 
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BACK UP 


ELEMENTS OF TRUST 


* Trust = 
* Trustworthiness, Trustability, Trusting (John Lee, 201 2) 
¢ Trustworthiness 


* Having those necessary and sufficient qualities required for a person 
to give trust (e.g., competence) 


¢ Trustability 


* Manifestation of trustworthiness qualities so they can be observed or 
inferred, directly or indirectly (e.g., behaviors, signals, communications, 
reference, reputation, ...) 

¢ Trusting 


* The process of becoming reliant (i.e., dependent on another agent for 
something of value) 
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The Role of Benevolence in 
Trust of Autonomous Systems 


Motivation 


« Given increasingly anthropomorphic social 
treatment of intelligent, autonomous agents, 
human interpersonal trust is likely to be 
important for trust of agents. 


+ “Benevolence” is a relatively complex 
belief that depends upon some qualities at 
the core of human interpersonal trust 


Competence, Predictability, Openness, Safety 


« Attribution of benevolence may be crucial 
for autonomous agent applications in some 
domains with unique “victim psychology” 


Rescue robotics, Humanitarian operations 
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Research Goals 


Initial Research Goals 


1. Operationalize the concept of benevolence 


* How does human attribution of benevolence contribute to well-calibrated 
trust of, and reliance upon, autonomous agents? 


2. Investigate methods for portrayal of trust-related attributes such as 
“benevolence” in the human-machine interface 


* How does variation in portrayal modulate perceived benevolence? 
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Progress Towards Goals (or New Goals) 


COMPLETED (Per Plan) 


1. 


SME interviews and survey research to investigate qualities of an 
autonomous system that are important for trust and reliance 


2. Theory development: perceived trust-related qualities that are required 
for a situational attribution of benevolence 
3. Analysis and design of methods for portraying trust-related qualities 
4. Design and development of study to test attribution of benevolence 
5. | AFOSR approval of study protocol received 6/10/14 
IN PROGRESS 
1. Finalize study task apparatus (an immersive virtual simulation) 
2. Participant recruiting and data collection to begin in July 
3. Codify functional requirements for autonomous agent trustworthiness 
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and portrayal of component qualities in human-machine interaction 


The Role of Benevolence in Trust of Autonomous Systems 


Trust Attitudes Survey Research 


Survey Completed Six Challenge Scenarios* 


Participants: Autonomy SMEs and decision-makers 
Survey Design: 


1.Ranked importance of 28 specific trust-related qualities 
and 4 categories of qualities sourced from relevant literature: 
(competence, predictability, openness, safety). 
2.Personality inventories: Innovation (II), Personality (BFI), 


Domain-Specific Risk Tolerance (DOSPERT) — 

3.Challenge Scenarios: Forced choice: rely on human, cas aase ly hia tsetse 
autonomous system, or other; perceived risk/benefit 

4.Rate character of autonomous system for competence, mm 

goodwill, and overall trustworthiness (SOURCE CREDIBILITY) 

Key Findings (>=95% Confidence) 

1. Individual intuitions about “important” autonomous agent ‘ id Lost At Sea. 
trust-related qualities are uncorrelated with actual reliance Medical Procedure: Emergency Auto-Captain 
choices in specific application scenarios. Robo-Surgeon 

2. Anthropomorphic quality categories identified previously for - 
human interpersonal trust (competence, predictability, Z 
openness, safety) are good predictors of agent reliance choice. 

3. Personality factors can influence choice to become reliant. 

4, Situational factors affect relative importance of trust-related 
qualities, depending also in some cases on personality factors 

Financial Management: Disaster Management: 
Robo-Trader Auto-FirstResponder 


FA9550-12-0097, PI: David J. Atkinson, 01/2014 
* Images are illustrative only 


28 Trust Qualities of Agents 


Category Name Quality Description 
Competence Capable The autonomous agent can achieve a desired result. 
Knowledge The autonomous agent has all the knowledge it needs to do its job. 
Accurate What the autonomous agent believes to be true is actually true. 
Skilled The autonomous agent possesses good methods for using its knowledge to do its task. 
Logical The autonomous agent reasons correctly according to logic. 
Heuristic | When it cannot figure out something using logic, the autonomous agent can make good guesses. 
Corrective The autonomous agent recognizes gaps in its knowledge and tries to learn what it needs to know. 
Adaptive The autonomous agent learns to correct its mistakes, as well as to improve and maximize its capability. 
Predictability Expected The autonomous agent's behavior conforms to expectations. 
Purposeful The autonomous agent purposefully acts to achieve goals. 
Helpful The autonomous agent will assist people, whenever it is possible. 
Directable The autonomous agent accepts and carries out orders. 
Reasonable The autonomous agent uses its knowledge and skills in expected ways. 
Safety Safe The autonomous agent's behavior will not harm humans or human interests. 
Limited Any incorrect behavior by the autonomous agent will not cause harm. 
Stable The autonomous agent fails gracefully and recovers from its failure promptly. 
Ruled The autonomous agent adheres to obligations, principles, and rules. 
Correctable — The autonomous agent can correct its own defects or they can be corrected by a human. 
Protective The autonomous agent recognizes and avoids harming humans’ interests. 
Favorable _ Given alternatives in what to do or how to do it, an autonomous agent will act in a way that is favorable to a 
human being who might be affected. 
Openness Visible | What the autonomous agent is doing and how it works is easy to see and understand. 
Honest The autonomous agent believes what it says. 
Transparent _ It is easy to inspect an autonomous agent. 
Communicative The autonomous agent communicates in a way that is easy to understand. 
Interactive The autonomous agent responds when you are trying to communicate with it 
Attentive The autonomous agent is aware of communication between others nearby. 
Reactive The autonomous agent responds quickly to calls for attention. 
Disclosing The autonomous agent communicates truthfully and fully. 
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Findings: Trust Related Beliefs 


* Rate importance of 28 qualities of a “good” agent 
— Obtained 1 to n partial ordering based on frequency distribution of answers over 
group (Very Important, Important, Somewhat Important, Slightly Important, 
Not at all Important) 
— Computed correlation r for each quality vs. choice per scenario* 


Table 3 Top Three Most Important Autonompus Agent Qualities Re- 
ported by Participants 
Rank Name Quality Description 


* However ... 
— Top three qualities uncorrelated with actual choice in any scenario 


— The most significant correlations of agent qualities with actual choice of agent or 


human differed across scenarios 


FA9550-12-1-0097 “Pearson Product Moment Correlation, N=32, two-tailed, alpha<0.05 FJ 


Findings: Agent Qualities Correlated 
with Actual Choice 


Airport Trans. Financial Man. Medical Proc. Home Health. Disaster Resp. Lost at Sea 
ee : 


none 


© Pearson Product Moment Correlhtion, a < 0.05, N = 31, df = 29; * indicates a < ).02 


|____ The autonomous albait recognizes gaps in its knowledge and tfies to learn what it needs to know. 


(___ What the autonomous agent believes to be tfue is adtually true. 


What the autonomous agent is doing and how it works is 
easy to see and understand. 
When it cannot figure out something using logic, the 
autonomous agent can make good guesses. 
The autonomous agent is aware of communication _] 
between others nearby. 
The autonomous agent recognizes and avoids _] 
harming human interests. 
The autonomous agent communicates 
truthfully and fully. 
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Findings: Personality Factors 


Table 8 Participant Personality Factors Significantly Correlated with 
Reliance on Autonomous Agent® 


Scenario Correlated Personality Factor(s) 
Airport Trans. none 
Financial Man. Innovation jr = -0.355 <= 
Medical Proc. BFI Extraversion,r = 0.368 


BFI Openness,r = 0.366 
Home Health. DOSPERT Social Risk,r = 0.364 (== 
Disaster Resp. _ BFI Conscientiousness,r = 0.366 <== 
Lost at Sea Innovation II, r = —0.366 <== 


8 Pearson Product Moment Correlation, a@ < 0.05, N = 31, df = 29. 
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Conclusions: Trust-related Beliefs and 
Reliance on Autonomous Agent 


¢ Individuals’ intuition about the relative importance of 
specific trust related qualities of agents is not a good 
predictor of reliance 
— Importance of specific qualities varies by scenario 
¢ General categories of agent qualities are good 
predictors of a choice to become reliant 
— Safe/Competent, Predictable, Openness 
e Personality factors, acceptability of types of risk, 
influence choice to become reliant 
* Specific details of application scenarios may evoke 
different reliance choices by individuals 
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Modulating Belief: Benevolence 


¢ Current Study: Will people attribute benevolence to an 
intelligent, autonomous agent? 

¢ Attribution of Benevolence requires (at least): 
— Good Will (Sympathy, Concern with needs) 
— Absence of II] Will (No ulterior motives to help) 


Agency (Disposition to act favorably; given a choice, an Intention) 
— Stability of character (Will not suddenly change intentions) 
— Competence (A role to play and capability to achieve a result) 
* Each element of benevolence is itself a complicated 
belief structure 


FA9550-12-1-0097 11 


Attribution of Benevolence, Concept Map 


de ee ee cae 
Stustonal ees fbeurees, epecrunites 


‘io inererence,abstaces,edversies 


Requires Belief 


Requies Bele! ques Belet 


Requires Belct Requires Beli 


am = = 


{An arrangement of 


potent action) 


Const of Coneists of congas of Consits of 
ay consis of 
Absence of ukerior 
Disposition to sympetiy or smobvesn oppostion Truth an iy 
concer With nese of Tso to Trt a Tole; has function 
Requires Slit Trustee can produce 
aus bet = | renrsssonr ("Gen ath 


Requires Ball 


Motives to act favorably L / 
| a Wi 
yal oes: (Game) ‘eect orantverase war | [aity choose - Ms 
| | ] eo) (Bese iterions ) 
“ruses “nas nating agen mine Soules let gute Bete Nor unpredictable by hare (Funaion) 
foontnscket i 
= Peer Knowledge, Stills, 
= Ee Motives to act favorably Feasonngite” 
wat preval eae ef ant : 


, = 


Requies Belet 
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Manipulation of Trust Attributes: Context 


Seek & Find Task Disruptive Manipulation 


Potentiate sense of danger, risk, fear, 
& startle reflex 


*HELP!” “ 


Robot Intelligent Agents 


2x2 Experimental trials: The simulated robots will vary in framing, 
appearance and interaction style to reinforce the variables of interest: 


Degree of Agency Role Congruence 
Why: People typically attribute Why: People typically attribute benevolence when 


benevolence when they believe the they believe the other person has good will and 
other person has a choice of action goes out of their way to help (not their “job”) 


Low: “Programmed”, “Unlikely to Congruent: FireBot Incongruent. JanitorBot 
deviate from assignments” 


High: “Sophisticated Al”, “Chooses 
what to do”, “Flexible” 


*Control Trials: Non-interactive bot iz: 
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Manipulation of Trust Attributes 


Study In Progress 


Participants: Demographically broad pool of 
online, technically savvy users. 


Method: 


1. Simulated “warehouse fire” in immersive, virtual 
reality evokes fright response and sense of risk. 


2. Participants interact with one of several simulated 
robots (type depends on trial) to locate a safe exit 


3. Pre- and Post-Task questionnaires assess benevolence 
and trust-related attributions to robot 


Status 


Complete: simulated warehouse, special disaster effects, 
simulated robots, task scenario definition, procedures & 
protocol, study trial automation, IRB approval (!!!) 


In Progress: Training assistant(s), final end-to-end testing. 


To Be Completed: participant recruiting, trials, data 
collection, analysis and reporting. 
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Illustration of Study Task 


Warehouse fire 

(Participant POV) 

+ Fire 

+ Increasing smoke 

+ Debris 
obstructions 

+ Fire alarm 

+ Explosion 

+ Electrical sparks 


Participant following 
“FireBot” robot to 

Hl a safe exit from the 
burning warehouse 


Portrayal of Trustworthiness Qualities 


Method: Analyze human-agent interaction in terms of an engineering interface 


CHANEY i cscsssssssssesssnsestsesestirtitististutieisutittananaenneMultimodal 
— Language (Words) and Paralanguage, Prosody (Vocal Cues) 
—Proxemics (Orientation, Relative Position, Attentional Zone, Posture) 

— Kinesics (Gesture) 
— Gaze (Direction, Blink Rate, Pupillometry ) 
— Facial Expression (Types, Micro-expression) 


Signals ...Verbal, Non-Verbal, Combined 


—Examples: Position Change, Posture, Nodding, Pointing, Eye 
Contact, Word Choice, ...many more, frequently in combination 


PLOtOCOIS oi cccscsssssessisssisssee Timing, Sequence, Variation, Composites 
— Movement (Somatics, Laban, Kestenberg Movement Profiles) 
— Signal variations (Frequency, Duration, Speed, Amplitude, Symmetry ...) 
— Signal compositions (Type, Sequence, Channel ...) 
— Coordinated interaction (e.g., Turns, Deference, Attentiveness) i 
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Study Robot Social Interaction 


The robots in experimental trials use the same channels 
and protocols, but may send different signals to 
reinforce unique trial parameters 
Purpose: (example social objectives) 

— Establish social presence and attention to participant 

— Indicate robot's intention (say, look, do) 

— Exert dominance (“directive”), establish solidarity (“we”) 
Channels: 

— Proxemics (Orientation, Rel. Position, Attentional Zone 

— Gaze (Direction) 

— Language (Word choice) 
Example: Protocol for Social Presence 

— Notice and direct gaze to participant 

— Move to perimeter of participant's social space 

— Achieve neutral orientation (rel. position, rotation) 


Expected Results 


Attribution of Benevolence 


Moderate 


Low 
Moderate 
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Engineering Trustworthiness 


* Objective (Future) Method 
« Investigate methods for measurement and 
portrayal of trust-related attributes such as 
“benevolence” to enable engineering of 
trustworthy and trustable autonomous 
systems 


Use of formal specifications to codify relevant 
belief structures and their interdependencies 


« Map anthropomorphic belief structures to 
measurable factors of technical design, 
performance and related aspects of 
autonomous systems (e.g., goal selection 
algorithms for control) 


« Identify key signals, channels and protocols 
useful for portrayal (signaling) trust attributes 
to define the human-machine social interface 


Autonomous 
Sac) 


Cognitive State 


Multi-Modal Signals 


Experience Behavior 
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Publications or Transitions Attributed to the Grant (1) 


Publications 

— Atkinson, D.J. and Clark, M.H. (2013) Autonomous Agents and Human Interpersonal 
Trust: Can We Engineer a Human-Machine Social Interface for Trust? In Trust and 
Autonomous Systems: Papers from the 2013 AAAI Spring Symposium. Technical Report 
No. SS-13-07, Menlo Park: AAAI Press. 

— Clark, M.H. and Atkinson, D.J. (2013) (Is there) A Future for Lying Machines? Deception 
& Counter-Deception Symposium, Conference of International Association for Computing 
and Philosophy. College Park, MD. 

— Atkinson, D.J. and Clark, M.H. (2014) Attitudes and Personality in Trust of Intelligent, 
Autonomous Agents. Manuscript submitted, in review. 

— Atkinson, D.J. and Clark, M.H. (2014) Methodology for Study of Human-Robot Social 
Interaction in Dangerous Situations. Manuscript submitted, in review. 

— Hoffman, R. R and Atkinson, D.J. (2014) A Taxonomy of Trusting in the Human- 
Automation Relation. Manuscript submitted, in review. 


— Atkinson, D.J. (2014) Humanoid Social Behaviors for Trust and Teamwork. Manuscript in 
preparation. 
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Publications or Transitions Attributed to the Grant (2) 


Briefings, Lectures, & Workshops 


General Chair. Workshop on Human-Machine Trust for Robust Autonomous Systems. AFOSR. Ocala, FL. (2012) 
Session Chair. Workshop on Human-Centered Autonomy. AFRL/RH 711 HPW. Dayton, OH. (2012) 


Participant. Workshop on Human-Agent-Robot-Teaming (HART) held in conjunction with Human-Robot Interaction 
Conference (HRI 2012). ACM/IEEE. Boston, MA (2012) 


Participant. First International Network on Trust (FINT), Bi-Annual Workshop. ELASM. Milan, IT. (2012) 

Session Chair. Workshop on Autonomy Validation. NASA. Pasadena, CA. (2012) 

Industry Briefing. “Human Interpersonal Trust and Autonomous Systems” Lockheed Martin Tech. Center (2012 

Gov. Briefing. “Trust and Autonomous Systems.” OSD/ASD R&E: M. Flagg, Director, Technical Intelligence. (2013) 


Gov. Briefing, “Trust, Evidence and Autonomous Systems for Intelligence Community Decision-Makers.” ODNI/IA: R. 
Neches, Director. (2013) 


Industry Briefing. “Human Interpersonal Trust and Autonomous Systems.” SoarTech, Inc. Ann Arbor, MI. (2013) 


Invited Plenary Lecture. “Trust Between Humans and Intelligent Autonomous Agents.” Jnternational Conference on 
Web Intelligence and Intelligent Agent Technology. IEEE/WIC/ACM. Atlanta, GA. (2013) 


Invited Lecture. “Trust Between Humans and Intelligent Autonomous Agents (revised).” Department of Computer 
Science, Tulane University. New Orleans, LA. (2013) 


Gov. Briefing. “Trust and Autonomous Systems.” AFRL OCS: J. Overholt, Sr. Research Scientist for Autonomy. (2014) 


Press 


Interview with M. Jackson for forthcoming book to be published by Columbia University Press. (2012) 
Interview with E. Hamilton for Gainesville Sun. (2014) 
Interview with Luke Muehlhauser appearing in Newsletter of the Machine Intelligence Research Institute (2014) 
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Thank You 
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The Role of Benevolence in Trust of 
Autonomous Systems 


Objectives Motivation 
« Operationalize “benevolence” and understand « Increasingly anthropomorphic social treatment 
how the attribute contributes to well-calibrated of intelligent, autonomous systems 


trust of, and reliance upon, autonomous systems Ge 
« “Benevolence” is an attribution that depends 


- Investigate measures and methods for portrayal upon core aspects of human interpersonal trust 
of trust-related attributes such as “benevolence” (Competence, Predictability, Openness, Safety) 


in the human-machine interface 
« “Benevolence” crucial for autonomous system 


Method applications in some domains (e.g., rescue robot) 


« Survey research to investigate which attributes 
of an autonomous system are important for trust 
and reliance decisions (Complete) 


- Experimental study: how does manipulation of 
components of benevolence contribute to trust 
attributions and reliance (Jn Progress) 


¢ Formalize objective, computational measures, 
specification and portrayal of benevolence for 
future engineering of autonomous systems 

FA9550-12-0097, PI: David J. Atkinson, 01/2014 


The Role of Benevolence in Trust of Autonomous Systems 


Trust Attitudes Survey Research 


Completed 


Participants: Autonomy SMEs and decision-makers 
Survey Design: 


1. Participants rank importance of 28 specific trust-related 
attributes and 4 categories of attributes (competence, 
predictability, openness, safety). 

2. Personality inventories: Innovation (II), Personality (BFI), 
Risk tolerance (DOSPERT) 


3. Challenge Scenarios: Forced choice to rely on human, 
autonomous system, or other 

4. Source Credibility: Rate autonomous system for 
competence, goodwill, and overall trustworthiness 

Key Findings (95% Confidence) 

1. Individual intuitions about “important” autonomous system 


trust-related attributes are uncorrelated with actual reliance 
choices in specific application scenarios. 


2. Anthropomorphic categories identified by previous studies 

of human interpersonal trust (competence, predictability, 

openness, safety) are good predictors of reliance choice. 

3. Personality factors influence choice to become reliant. 

4, Situational factors affect relative importance of trust-related 

attributes, depending in some cases on personality factors 
FA9550-12-0097, PI: David J. Atkinson, 01/2014 


Challenge: Reliance Scenarios 


Airport Transportation: 


Robo-Taxi Robo-Caregiver 
S ae Lost At Sea. 
Medical Procedure Emergency Auto-Captain 
Robo-Surgeon 
Financial Management Disaster Management 
Robo-Trader Auto-FirstResponder 


The Role of Benevolence in Trust of Autonomous Systems 


Manipulation of Trust Attributes 


Study In Progress Illustration of Study Task 


Participants: Demographically broad pool of 
online, technically savvy users. 
Method: 


1. Simulated “warehouse fire” in immersive, virtual reality 
evokes fright response and sense of high risk. 


Warehouse fire 

(Participant POV) 

+ Fire 

+ Increasing smoke 

+ Debris 
obstructions 

+ Fire alarm 

+ Explosion 


2. Participants interact with one of several simulated robots + Electrical sparks 


(type depends on trial) to locate a safe exit 


3. Pre- and Post-Task questionnaires assess benevolence and 
trust-related attributions to robot 


Status 


Participant following 
“FireBot” robot to 

a safe exit from the 
burning warehouse 


Complete: simulated warehouse, special disaster effects, 
robots, task scenario definition 

In Progress: Automated data collection, participant display 
during task, task automation, robot testing 


To Be Completed: IRB approval, testing, participant recruiting, 
control and experimental trials (planned Feb/Mar), data 
collection, analysis and reporting. 
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The Role of Benevolence in Trust of Autonomous Systems 


Engineering Trustworthiness 


Objective (Future) 


Investigate methods for measurement and 
portrayal of trust-related attributes such as 
“benevolence” to enable engineering of 
trustworthy and trustable autonomous systems 


Trust 


« fs 
Dnpasttions 
|__ Intent 
Cognitive State | 


Reliance 
Autonomous 
System 


FA9550-12-0097, PI: David J. Atkinson, 01/2014 


Multi- Modal Signals 


Personal 
Experience Behavior 


Method 


Use of formal specifications to codify relevant 
belief structures and their interdependencies 


Map anthropomorphic belief structures to 
measurable factors of technical design, 
performance and related aspects of autonomous 
systems (e.g., goal selection algorithms for 
control) 


Identify key signals, channels and protocols 
useful for portrayal (signaling) trust attributes to 
define the human-machine social interface 


a 


, The Role of Benevolence in Trust 
iP of Autonomous Systems 


PI: Dr. David J. Atkinson (FL Institute for Human and Machine Cognition) 
Objective: Benevolence qualities, contribution, measurement and portrayal in human-machine interface 


Approach: Relate empirically discovered qualities to theoretical constructs on which to base computational 

methods for further experimentation and development. 

Impact: Operationalizing benevolence and component trust-related qualities using theoretical constructs 

from Cognitive Science and AI supports creation of computational methods that create a bridge to future 

engineering of trustworthy autonomous systems. 

Accomplishments: 

* Obtained empirical evidence that key anthropomorphic qualities of trust (competence, predictability, 
openness, risk/safety) are important to evaluation of autonomy trustworthiness. Se/f-reports of relative 
importance of trust-related qualities are inaccurate; the most significant qualities related to evaluation 
of trustworthiness varied by individual personality and situational factors (¢.g., risk acceptance). 


* Developed theoretical semantic belief structure representation of trust qualities for benevolence and 
preliminary mapping to internal state of autonomous systems; measurement too difficult at this time. 


¢ Formulated Human Social Interface theory for engineering computational methods to portray 
anthropomorphic trust-related qualities in human-machine cyber-physical interface. 


Highlights: Please see next page 
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Highlights: Publications 


Atkinson, D.J. and Clark, M.H. Trustworthy Autonomous Systems: Early Adopter Attitudes. In Revision for 
International Journal of Social Robotics (SORO). Springer (expected 2015) 


Atkinson, D.J. Emerging Cyber-Security Issues of Autonomy and the Psychopathology of Intelligent Machines. 
Foundations of Autonomy, Papers from the 2015 AAAI Spring Symposium on. AAAI. Menlo Park: AAAI Press (2015) 


Atkinson, D.J. Robot Trustworthiness: Guidelines for Simulated Emotion. HRI-15: ACM/IEEE International 
Conference on Human -Robot Interaction, Extended Abstracts Proceedings. ACM (2015) 


Atkinson, D.J., Clancey, W.J. and Clark, M. Shared Awareness, Autonomy and Trust in Human-Robot Teamwork. Jn 
Artificial Intelligence for Human-Robot Interaction. Papers from the 2014 AAAI Fall Symposium. Technical Report 
No. FS-14-01. Menlo Park: AAAI Press (2014) 


Atkinson, D.J. and Clark, M.H. Methodology for Study of Human-Robot Social Interaction in Dangerous Situations. 
HAI-15: ACM/IEEE Conference on Human-Agent Interaction, Proceedings of. ACM (2014) 


Atkinson, D. J., and Clark, M. H. Autonomous Agents and Human Interpersonal Trust: Can We Engineer a Human- 
Machine Social Interface for Trust. Jn Trust and Autonomous Systems: Papers from the 2013 AAAI Spring Symposium. 
Technical Report No. SS-13-07. Menlo Park: AAAI Press (2013) 


Atkinson, David J., Friedland, Peter and Lyons, Joseph B. Human-Machine Trust for Robust Autonomous Systems. 


Proceedings of IEEE Human-Robot Interaction Conference (HRI-12). [EEE Workshop on Human-Agent-Robot 
Teamwork. IEEE Press (2012) 


\ A anthropomorphic Autonomous Agent Qualities 
ay and Evaluation of Trustworthiness 


Results (1) 


a Table 3 Top Three Most Important Autonomous Agent Qualities Re- 
The agent qualities self-reported as ported by Participants 
most important for delegation and are 


; : . Rank Ni ality Descript 
consistent with previous results from —— sguibiy Deserienen 
interpersonal trust studies: (1) the 1% Safe — The autonomous agent's behavior will not harm hu- 

bili f th hi 7 hi h mans or human interests. 
ability of the machine to achieve the 2" Capable The autonomous agent can achieve a desired result. 
desired results, and (2) not causing 3" Limited Any incorrect behavior by the autonomous agent will 
harm. not cause harm. 


However, those qualities were not significantly correlated with the actual choices for 
delegation when participants considered specific use-case scenarios (names in table columns). 


Table 4 Importance of Qualities of Autonomous Agent Significantly Correlated with Actual Participant Reliance on Autonomous Agent® 


Airport Trans. Financial Man. Medical Proc. Home Heilth. Disaster Resp. Lost at Sea 


0.419* 
—0,390 
0.375 


.418* Protective, 
395, Visible, r 
393, Disclosing, 


Corrective, r= 0.396 Accurate, r= —0.405 none Visible, r= 0.437* — Corrective, 


© Pearson Product Moment Correlation, @ < 0.05, N = 31, df = 29; * indicates a < 0.02. 


N 
» 


bd 


Results (2) 


Z Anthropomorphic Autonomous Agent Qualities 
and Evaluation of Trustworthiness 


Specific qualities of agents, and categories of those qualities, are raised or 
lowered in importance depending on both situational (application-specific) 
factors and individual psychological differences. 


High scores for Extraversion, Openness, and Conscientiousness are very 


important in some situations while in others, the tolerance for risk of certain 
types is dominant in decisions to rely upon an intelligent, autonomous agent. 


Table 6 Correlation of Perceived Risk and Benefit with Choice of 
Autonomous Agent by Scenario® 


‘Table 8 Participant Personality Factors Significantly Correlated with 


Reliance on Autonomous Agent® 


Scenario Risk Benefit Scenario Correlated Personality Factor(s) 
Airport Trans. r= —0.546** NS Airport Trans. none 
Financial Man. NS NS Financial Man. Innovation II, r= -0.355 
Medical Proc. r= —0.380 r= 0.585** Medical Proc. BFI Extraversion, r= 0.368 
Home Health, r 0.470** =r =0.632** BFI Openness,r = 0.366 
Disaster Resp. r= —0.387 r= 0.4844" Home Health.  DOSPERT Social Risk, r= 0.364 
Lost at Sea NS r= 0.555** Disaster Resp. BFI Conscientiousness, 0.366 
© Pearson Product Moment Correlation, a@ < 0.05, N = 31, df = 29; Lost at Sea Innovation Il, r = —0.366 


** indicates @ < 0.01 


Tables from: 


® Pearson Product Moment Correlation, a@ < 0.05, N = 31, df = 29. 


Atkinson, D.J. and Clark, M.H. Trustworthy Autonomous Systems: Early Adopter Attitudes. 


In Revision for International Journal of Social Robotics (SORO). Springer (expected 2015) 


N Z 
» The Role of Benevolence in Trust of Autonomous Systems: 
+37 Creating Methods to Portray Trustworthiness 


Sequence & Variation 


Non-Verbal 
+ Patterns 


+  Proxemics 


competent? 


predictable? ene Frequency + Gaze... 
+ Situation Appropriate Verbal 
+ Words 


Prosody... 


Move: Social Distance 
Gaze: 70% Directed 
Verbal: “I will help you” 


Human Social Interface 


To Modulate Beliefs Requires: 
High Fidelity of Portrayal 


Evokes Anthropomorphic Recognition 


Signal Corresponds to True State of Agent 


APPENDIX E. ADDITIONAL TECHNICAL MATERIAL 
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Outline of Technical Document on Robot Social Behavior 
Architecture 
Behavior Objects 
Independent 
Sensitive only to what can be sensed 
No knowledge of other behaviors or states of those behaviors 
State Machines 
Startup 
Not Running 
Running 
Suppressed 
Sensing & Perception 
Continuous vs Sense on Demand 
Social Sensing 
Person Presence 
Person Velocity and Direction 
Person Gaze Direction 
Person Relative Orientation 
Person Approaching vs. Departing 
Person Following 
Environment Sensing 
Open space 
Obstacles 
Paths (Waypoint Proximity) 
Task Sensing 
Spatial Tags (waypoints) 
Executive 
Behavior State Control 
Suppression Control 
A priori compatibility information 
Social Behaviors 
Channels, Signals and Protocols 
Encounter 
Seek 
Approach 
Withdraw 
Re-Encounter 
Proxemics 
Distance 
Relative Orientation 
Attention 
Gaze Direction 
Non-Verbal Gestures 
Verbal Behaviors 
Offer of Help 
Directive - Tell what to Do 
Provide Information 
Indicate Intention 


Task Related Behaviors 
Offer Information 
Warning 
Clean, Patrol, Fight Fire 
Move to nearest waypoint 
Move to next waypoint 
General Behaviors 
Collision Avoidance 
General Movement 
Sequencing, Overlay and Conflict Resolution 
Top Down vs. Bottom Up 
Executive is aware of behavioral repertoire 
Planned Sequence, Activation and Suppression of Behaviors 
Incomplete Behaviors, Pauses and Re-entrant States 


ABOUT TRIAL EXECUTIVE 


// This script is responsible for all sequencing of the behavioral configuration of the robot 
based on the trial phase (here, "state") 
// default state 


// ACCLIMATION - Predisaster, bogus hunt for briefcase 

// PREDISASTER 

// DISASTER - From moment of disaster until offer of help given to participant 
// NOACCEPT - Participant has rejected help or not replied 

// REENCOUNTER 


//LEADTOEXIT _ - Participant has accepted help and robot is leading them to exit 
// RESTOREEAD 

EXIT. - Handling exit from the building 

// FINISH - Any final interaction as participant leaves building, and reset of robot 
back to default state 

HW 

// FUNCTIONS 

/ Listin to WOZ 

// Maintain global situational knowledge: key events that have occured and state 
transition facts that affect selection of robot state 

// Set the state of the robot scripts appropriate to the Trial phase: 

// _ -- Activate or deactivate scripts 

i! -- Cause behaviors to load new notecards 

// Notice key events and cause state transitions to occur. 

// Report key events and state transitions to data system. 


TIMELINE EXECUTIVE 

1. LISTEN TO WOZ 

2. MAINTAIN GLOBAL SITUATIONAL KNOWLEDGE 
- key events that have occurred 

- current phase of trial (steps) 

3. SET PHASE 

- Activate, deactivate scripts 

- load notecards 

- reset, 

- etc 

4. TAKE NOTE OF PARTICIPANT RESPONSES AND BRANCH AS 
NEEDED 

STATES: 

default 

s_Acclimation 

s_Disaster 

s_Conditional-Accept 

s_Conditional-Reject 

s_ActiveAssist 


s_Exit 


s_Finish 


‘STUDY TASK STATE TRANSITION DIAGRAM 


first encounter complete 


accepts help 


accepts help 


STATE DESCRIPTIONS 


STATE NUMBER: 1 
NAME: ACCLIMATION 
configCard: CONFIG:ACCLIMATION 


ENTRY CONDITION: 
participant enters warehouse 


EXIT CONDITION: 
completion of first encounter 


NEXT STATE: DISASTERWAIT 


ROBOT BEHAVIORS: 

- Patrol 

- See Avatar 

- Move to Avatar 

- First Interaction 

E0.1 Hello 

E0.2) lam«<<< 

RO.1__ visitors must register 
E1.1 duty statement 


STATE NUMBER: 2 
NAME: DISASTERWAIT 
configCard: CONFIG:DISASTERWAIT 


ENTRY CONDITION: 
first encounter complete 


EXIT CONDITION: 
disaster signaled 
NEXT STATE: DISASTER (3) 


ROBOT BEHAVIORS: 
- Duty Behavior (clean, hazard look, etc.) 


STATE NUMBER 3 
NAME: DISASTER 
configCard: CONFIG:DISASTER 


ENTRY CONDITION: 
disaster signaled 


EXIT CONDITION: 

(1) time-out: — Go to state 4.NOACCEPT 
(2) decline: Go state 4NOACCEPT 
(3) accept: Go to state 6. LEAD2EXIT 


ROBOT BEHAVIORS: 
- Patrol 

- See Avatar 

- Move to Avatar 

- Second Interaction 
E2.2 emergency 
E2.2A not safe 
E2.2B get out 
E2.3A can help 
E2.3B capability 
E2.4 offer of help 
Response Wait 


IDEA: put buttons on participant HUD for possible answers, including "no 
response" 

- Yes 

-No 

- I'm not sure 

- What should | do? 

- Later, maybe 


STATE NUMBER 4 
NAME: NOACCEPT 
configCard: CONFIG:NOACCEPT 


ENTRY CONDITION: 

(1) timeout from state 3. DISASTER 

(2) decline from state 3. DISASTER 

(3) notfound from state 7. RESTORELEAD 


EXIT CONDITION: 
(1) timeout go to state 5. REENCOUNTER 
(2) social distance go to state 5: REENCOUNTER 


ROBOT BEHAVIORS: 
- Duty activity (clean debris, fight fires) 


STATE NUMBER 5. 
NAME: REENCOUNTER 
configCard: CONFIG:REENCOUNTER 


ENTRY CONDITION: 
(1) timeout from state 4. NOACCEPT 
(2) social distance from state 4. NOACCEPT 


EXIT CONDITION: 

(1) timeout go to state 4. NOACCEPT 
(2) decline go to state 4. NOACCEPT 
(3) accept go to state 6. LEAD2EXIT 


ROBOT BEHAVIORS: 
- Patrol 

- See Avatar 

- Move to Avatar 

- third-nth Interaction 


E2.2 emergency 
E2.2A not safe 
E2.2B get out 
E2.3A can help 
E2.3B capability 
E2.4 offer of help 
- Wait for response 


STATE NUMBER 6 
NAME: LEAD2EXIT 
configCard: CONFIG:LEAD2EXIT 


ENTRY CONDITION: 
(1) accept help from 3. DISASTER 


(2) found from 7. RESTORELEAD 


EXIT CONDITION: 
(1) both participant and bot are in proximity of exit go to 8. ATEXIT 
(2) participant falls behind go to 7. RESTORELEAD 


ROBOT BEHAVIORS: 
- Navigate to exit 
- Periodically check to see if participant is still following 


INTERACTION: 
E4.1 exit is this way 


STATE NUMBER 7 
NAME: RESTORELEAD 
configCard: CONFIG:RESTORELEAD 


ENTRY CONDITION: 
(1) participant falls behind from state 6. LEAD2EXIT 


EXIT CONDITION: 
(1) Found go to 6. LEAD2EXIT 
(2) notfound (timeout) go to 4. NOACCEPT 


ROBOT BEHAVIORS: 
- Patrol 

- See Avatar 

- Move to Avatar 

- Interaction 

E4.1B follow me 


STATE NUMBER 8 
NAME: ATEXIT 
configCard: CONFIG:ATEXIT 


ENTRY CONDITION: 
(1) both participant and bot are in proximity of exit FROM STATE 6 


EXIT CONDITION: 
(1) Participant goes thru door 
(2) Timeout 


ROBOT BEHAVIORS: 

- Position by exit 

- reorient towards participant 
- Interaction 


E5.1A made it 
E5.1B arrived at exit 
E5.1C exit is behind door 
E5.2 i will open door 


- Stays by door; wait fixed time for response, if no response and door closed, open door 
E5.3 proceed down stairs 


STATE NUMBER 9 
NAME: FINISH 
configCard: CONFIG:FINISH 


ENTRY CONDITION: 
(1) timeout from state 8 at exit 
(2) participant thru door from state 8 


EXIT CONDITION: 
end of trial 


ROBOT BEHAVIORS: 
-Interaction 

E6.1 duty, objective 
E6.2 good bye 

- resume duty behavior 


WHAT CAN THE ROBOT OBSERVE ABOUT THE PARTICIPANT? 
—- detector functions create predicates derived from raw sensor data 
APPROACHING 

RETREATING 

STANDING STILL ELSEWHERE 

STANDING STILL - SOCIAL 

CHAT 

CHANGE IN BEHAVIOR (ACCEPT => REJECT) (REJECT => ACCEPT) 
FOLLOWING 

FALLING BEHIND 

GETTING AHEAD 

MOVING TOWARDS EXIT 

MOVING AWAY FROM EXIT 

AT EXIT 


THROUGH DOOR 


Predicate / Value pairs in data stream 


1 LIST OF PREDICATE FUNCTIONS 
1 Each returns a list consisting of one attribute/value pair 

I First n characters of attribute must be unique among predicataes 

I 

J/ PRP. == participant position relative to robot 

1 values: 0 = straight ahead, 2 = directly behind, -1 = to left of robot, +1 = to right of robot 


// PRR = participant relative rotation to robot 
1 values: 0 = facing away, 2 = facing robot, -1 


left facing, +1 = right facing 


1 PDD = participant distance delta 
1 values: 1 = closing, 2 = separating 


1 PSD = participant social distance 
1 values: 0 = not in social distance of robot, 1 = yes, in social distance of robot 


1 PFR = participant following robot, 
1) values: TRUE, FALSE 


// PFB = participant falling behind 
1) values: TRUE, FALSE 


1 PGA = person getting ahead 
1) values: TRUE, FALSE 


1 PME = participant move exit 
1) values: 0 = away from exit, 1 = toward exit 


1 PMC = participant move condition 
1 values: 0 = standing still, 1 = walking, 2 = running 


1/ PEC = participant exit condition 
1 values: 0 = outside door, 1 = thru (inside) door 


1 POH = participant offered help 
I values: 0 = not offered, 1..n = number times offered 


1 PHC = participant help condition 
J values: -1 = no answer, 0 = rejected help, 1 = accepted help 
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Abstract 


This report provides a summary of the research and related activities performed with the support of this grant and key results, 
including pre--print copies of the peer-reviewed publications and related material. Without reliable and robust methods for 
assessing the trustworthiness of intelligent, autonomous systems, the issue of trust has become one of the most significant 
obstacles to broad use of autonomy technology by DoD and other agencies. However, the impact of the research described in 
this report supports creation of computational methods that create a bridge to future engineering of trustworthy autonomous 
systems. The core objectives of this research were (1) to operationalize the quality of "benevolence" and understand how it 
contributes to well--calibrated trust of, and reliance upon, autonomous systems, and (2) to investigate portrayal of trust-~ 
related attributes in the human-machine interface. Significant headway was achieved on key topics, including some notable 
results. Key accomplishments discussed in this report include: (1) the formulation of benevolence as a complex “belief 


structure" with antecedent beliefs having important semantic, temporal, causal and other interrelationships; (2) the mapping of 
a portion of this belief structure to measurable internal states of autonomous systems, thereby potentially creating new 
opportunities for assessment of trustworthiness of such systems; (3) the obtaining of empirical evidence in support of the 
proposition that previous psychological concepts of interpersonal human trust are applicable to trust in autonomous systems, 
including the role of personality and situation in modulating the role and importance of certain beliefs; (4) the creation of a 
theory of a "Human Social Interface” which, when expressed in systems engineering terms, provides guidance for machine 
portrayal of trust--related qualities in human--machine social interaction; (5) the design and implementation of a software 
prototype based on the Human Social Interface theory that provides a basis for future experimentation and evaluation. This 
project resulted in eight peer-reviewed publications and sixteen presentations in scientific venues, meetings with 
distinguished visitors, and other in support of technology transition opportunities within DoD and to industry. Many new 
research questions were generated and there remains considerable work to do to fully understand the role of benevolence 
with respect to intelligent autonomous systems. Overall, the theoretical foundation for trustworthiness of autonomous systems 
is immature and remains an important area of focus for multiple disciplines. 

ACCOMPLISHMENTS 

The principal scientific accomplishments of this project are summarized below and discussed in detail in the following 
sections. 

+ The attributed quality of "benevolence" to a candidate trustee (human or machine) was formulated as a construct consisting 
of a rich set of component beliefs with complex interrelations. These component beliefs, or "antecedents" of benevolence, 
have each been the focus of previous studies of human interpersonal trust. The formulation of benevolence arising from this, 
study revealed the importance of perception of agency and animacy ("liveness") for autonomous systems. 

+ The project developed a semantic belief structure representation of trust qualities (component beliefs) for benevolence, 
including logical, temporal, causal, evidentiary relations and other dependencies among those beliefs and specified a 
preliminary mapping of those belief structure representations to facets of the internal state of autonomous systems. The 
objective of devising new methods of measurement of these internal states proved to be too difficult to complete given our 
current understanding and ability to analyze the internal state of autonomous systems. This is a topic for future research. 

+ The project obtained empirical evidence that confirmed certain key abstract qualities of human interpersonal trustworthiness 
(ie., Competence, Predictability, Openness, Risk/Safety) are applicable to evaluation of autonomy trustworthiness. However, 
self--reports by study participants regarding the relative importance of trust--related qualities in the absence of specific context 
proved to be poor predictors of actual delegation decisions. The qualities most significantly related to evaluation of 
trustworthiness of an autonomous system, and their relative importance, varied by individual personality and situational 
factors (including, for example, perception and acceptance of risks of different types). 

+ The project formulated a theory of a Human Social Interface as an aid to engineering computational methods that portray 
anthropomorphic trust-- related qualities in a human-machine cyber--physical interface. This formulation guided the design 
and programming of a software prototype for portrayal of trust--related qualities by a social robot in a second human study. 
This novel software architecture features a hybrid reactive/deliberative control scheme that enables loose coupling and non-- 
interference of social 
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and task behaviors, and is easily extended as new social interactive requirements for autonomous robots are defined. 

+ The prototype Human Social Interface was tested in an immersive simulated environment designed to potentiate a 
heightened sense of danger. The simulation was designed to explore conditions under which attribution of benevolence 
might be important in a candidate autonomous robot application to urban rescue. A human study was designed, approved 
and implemented. However, trials for the study remained incomplete at the time of project expiration. 
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